Appendix

Magic Syntax
Magic grammar is a feature in Jupyter Notebook that allows users to perform special operations through specific commands, usually starting with % or %% . Magic commands can be used to simplify common tasks and improve work efficiency.
Jupyter Notebook Native Magic Commands
Line magic commands usually start with % and act on the current line.
Example:
Magic Command
Description
Use Case
%run
Run the specified Python script or notebook file
%run example.ipynb
%pip
Install the specified Python package
%pip install <package name>
Cell magics usually start with %% and apply to the entire cell.
Example:
Magic Command
Description
Use Case
%%python
Execute the current cell in Python syntax (usually unnecessary, as Python is the default)
%%python
print('hello world')
%%markdown
Render Markdown text in the cell
%%markdown
# This is a title
## This is a secondary title
For more magic commands, see Magic Official Website.
WeData Notebook Special Magic Commands
Multilingual Switch
By default, Notebook uses Python grammar for code writing. If you want to switch the grammar of the current cell, you can add %%<language> at the beginning of the cell.
To execute the following magic code, you need to run the following initialization commands first:
%load_ext dlcmagic.kyuubikernel.magics.dlcenginemagics
%load_ext dlcmagic.pythonkernel.magics
Example:
Magic Command
Description
Use Case
%%py
Execute the current cell in PySpark grammar. You need to execute the following initialization commands before using it.
%%py
# Create sample data
data = [("Alice", 1), ("Bob", 2), ("Cathy", 3)]
columns = ["Name", "Id"]
# Create a DataFrame
df = spark.createDataFrame(data, columns)
# Display the DataFrame
df.show()
%%scala
Execute the current cell in Scala grammar.
%%scala
println("Hello, World!")
%%sql
Execute the current cell in Spark SQL grammar.
%%sql
SELECT * FROM example_table_1;
Note:
The special magic commands of WeData Notebook above are only applicable to use when connected to the DLC engine machine learning resource group.
DLC Utilities (Dlcutils)
dlcutils is a utility library provided by WeData based on the DLC engine, mainly used to simplify various operations in the Notebook environment, which can help users perform tasks related to data processing, parameter passing, environment configuration, etc.
Data Utility (Dlcutils.Notebook)
Function Name
Description
Use Case
summarize(df: Object): void
Calculate and display statistical metrics for DataFrame to understand the data structure.
Applicable to Python code, PySpark code.
# Create sample data
data = [("Alice", 1), ("Bob", 2), ("Cathy", 3)]columns = ["Name", "Id"]
# Create a DataFrame
df = spark.createDataFrame(data, columns)
# Overview of DataFrame
dlcutils.data.summarize(df)
Notebook Utility (Dlcutils.Notebook)
Function Name
Description
Use Case
exit(value: String): void
Exit the current notebook and print the specified return value, which can be used to pass the parameters of the notebook to downstream tasks.
dlcutils.notebook.exit('this is output  parameter values')
#  DLCNotebook exited: this is output  parameter values.
run(path: String, timeoutSeconds: int, arguments: Map): String
Run the notebook file
path: Specify the notebook file path
timeoutSeconds: Timeout Time
arguments: Variables
res = dlcutils.notebook.run("nb.ipynb", timeout_seconds=60, arguments={'key1':8, 'key2':'value'})
Parameters Utility (Dlcutils.Params)
Function Name
Description
Use Case
text(name: String, defaultValue: String, label: String): void
Set variable value
dlcutils.params.text(
  name='your_name_text',
  defaultValue='Ricky',
  label='Your name'
)
﻿
print(dlcutils.params.get("your_name_text"))
﻿
# Ricky
get(name: String): String
Get the specified variable
dlcutils.params.text(
  name='your_name_text',
  defaultValue='Ricky',
  label='Your name'
)
﻿
print(dlcutils.params.get("your_name_text"))
﻿
# Ricky
remove(name: String): void
Clear specified variable
dlcutils.params.remove('fruits_combobox')
removeAll(): void
Clear variables set in the current context
dlcutils.params.removeAll()
Note:
The dlcutils functions above are only applicable to use when connected to the DLC engine machine learning resource group.
MLFlow Function
MLFlow is an open-source machine learning platform that provides end-to-end support for the data science lifecycle, including experiment management, model version control, model deployment, and model monitoring.
Tencent Cloud WeData provides machine learning experiment management and model management capabilities based on MLFlow. If the MLFlow service is enabled in the Notebook workspace, you can record the parameters, indicators, and results of each experiment by calling MLFlow's related functions during the experiment, and view them in WeData Machine Learning Module > Experiment Management and Model Management, thereby achieving experiment tracking and reproducibility.
Common MLFlow functions include:
﻿MLFlow Function
Function Name
Function Feature and Usage
Experiment Management
mlflow.create_experiment(name)
Create a new experiment.
Ensure the uniqueness of the experiment name; if the experiment name already exists, create_experiment will raise an exception.
﻿
mlflow.set_experiment(name)
Set the current experiment.
It can be used directly for an existing experiment name to record parameters and indicators in subsequent runs.
If the specified experiment does not exist, a new experiment will be created automatically.
﻿
mlflow.start_run()
Start a new run.
Return a Run object representing the context of the current run.
start_run() is usually used with the with statement to ensure that end_run() is called automatically after the run ends.
Record parameters and indicators.
mlflow.log_param(key, value)
Record a parameter and its value.
key (str): Name of the parameter.
value (str, int, float): The value of the parameter. It can be a string, integer, or floating-point number.
﻿
mlflow.log_metric(key, value, step=None)
Record an indicator and its value.
﻿
mlflow.log_artifact(local_path, artifact_path=None)
Record local files or directories, such as the model's configuration file, data file, result file, etc.
local_path: The path of the local file or directory to be recorded;
artifact_path: The path on the MLflow server where the file or directory is stored.
Model Management
mlflow.sklearn.log_model(model, artifact_path)
Record Scikit-learn models.
﻿
mlflow.pyfunc.log_model(artifact_path, python_model)
Record a custom Python model.
﻿
mlflow.register_model(model_uri, name)
Register the model to the model registry. The model registry is the model management and version control feature provided by MLflow, which facilitates the sharing, deployment and management of models.
Model deployment
mlflow.pyfunc.serve(model_uri)
Deploy the model as a REST API service. It is used to start an HTTP server locally to provide prediction services for registered MLflow models. After the server is started, prediction can be made by sending data through HTTP Post requests.
model_uri: URI pointing to the registered model, which can be the URI in the model registry or the path of the recorded model.
For more MLFlow functions, see MLflow Overview.
Notebook Parameter Usage Example
Use project parameters in notebook
1. Go to Project Management > Parameter Settings interface, click Add to complete the creation of new project parameters.
﻿
2. If a parameter has already been defined in the project management parameters, for example, the parameter name is test_parameter and its value is 100, you can directly use the project parameter in the notebook.
# print project parameters  
print(dlcutils.params.get("test_parameter"))
# output 100
Use workflow parameters or task parameters in notebook
Because when debugging and running notebook files in the Notebook space, the notebook has not been associated with tasks and workflows, default values can be set in the test. Use the values configured on the workflow and task during Notebook periodic scheduling.
Workflow Parameters can be configured in the workflow general settings.
Scheduling Parameters can be configured in the task attributes.
For example, the task is configured with the task parameter task_test_param, and the workflow of the task is configured with workflow_test_param.
﻿
﻿
# get task_test_param value
# When testing and running in the notebook space, 
# default values need to be set because the notebook file has not yet been associated with a task.
﻿
try:
    task_test_param_value = dlcutils.params.get("task_test_param")
    if not task_test_param_value:  # If the obtained value is an empty string
        task_test_param_value = 'task_default_value'
except Exception:  # If the parameter cannot be obtained at all
    task_test_param_value = 'task_default_value'
﻿
print(f"Using toy value: {task_test_param_value}")
# get workflow_test_param value
# When testing and running in the notebook space, 
# default values need to be set because the notebook file has not yet been associated with a workflow.
try:
    workflow_test_param_value = dlcutils.params.get("workflow_test_param")
    if not workflow_test_param_value:  # If the obtained value is an empty string
        workflow_test_param_value = 'workflow_default_value'
except Exception:  # If no parameter is obtained at all
    workflow_test_param_value = 'workflow_default_value'
﻿
print(f"Using toy value: {workflow_test_param_value}")
Parameter transmission between notebooks
1. In the Notebook space, create two notebook files: parameter_test_up.ipynb and parameter_test_down.ipynb, with the following contents:
parameter_test_up.ipynb
# Exit the notebook and output parameters
dlcutils.notebook.exit('this is output  parameter values')
parameter_test_down.ipynb
# get task_input_param value
# When testing and running in the notebook space, 
# default values need to be set because the notebook file has not yet been associated with a task.
try:
    task_input_param = dlcutils.params.get("task_input_param")
    if not task_input_param:  # If the obtained value is an empty string
        task_input_param = 'task_input_default_value'
except Exception:  # If the parameter cannot be obtained at all
    task_input_param = 'task_input_default_value'
﻿
print(f"Using toy value: {task_input_param}")
2. At the same time, create two tasks in the orchestration space: notebook_upstream_01 and notebook_downstream_01, and select the above two notebook files respectively.
﻿
3. In the scheduling settings of the task notebook_upstream_01, set the Output the current task parameters task_output_param to $[0].
﻿
4. In the scheduling settings of the task notebook_downstream_01, set the output of the current task parameter task_output_param to $[0].
﻿
5. Finally, in the workflow debug run or scheduling run, you can view the output of notebook_downstream_01 as this is output parameter values.
﻿

Was this page helpful?

You can also Contact Sales or Submit a Ticket for help.

Yes

Magic Command	Description	Use Case
%run	Run the specified Python script or notebook file	%run example.ipynb
%pip	Install the specified Python package	%pip install <package name>

Function Name	Description	Use Case
exit(value: String): void	Exit the current notebook and print the specified return value, which can be used to pass the parameters of the notebook to downstream tasks.	dlcutils.notebook.exit('this is output parameter values') # DLCNotebook exited: this is output parameter values.
run(path: String, timeoutSeconds: int, arguments: Map): String	Run the notebook file path: Specify the notebook file path timeoutSeconds: Timeout Time arguments: Variables	res = dlcutils.notebook.run("nb.ipynb", timeout_seconds=60, arguments={'key1':8, 'key2':'value'})

MLFlow Function	Function Name	Function Feature and Usage
Experiment Management	mlflow.create_experiment(name)	Create a new experiment. Ensure the uniqueness of the experiment name; if the experiment name already exists, create_experiment will raise an exception.
		mlflow.set_experiment(name)	Set the current experiment. It can be used directly for an existing experiment name to record parameters and indicators in subsequent runs. If the specified experiment does not exist, a new experiment will be created automatically.
		mlflow.start_run()	Start a new run. Return a Run object representing the context of the current run. start_run() is usually used with the with statement to ensure that end_run() is called automatically after the run ends.
Record parameters and indicators.	mlflow.log_param(key, value)	Record a parameter and its value. key (str): Name of the parameter. value (str, int, float): The value of the parameter. It can be a string, integer, or floating-point number.
		mlflow.log_metric(key, value, step=None)	Record an indicator and its value.
		mlflow.log_artifact(local_path, artifact_path=None)	Record local files or directories, such as the model's configuration file, data file, result file, etc. local_path: The path of the local file or directory to be recorded; artifact_path: The path on the MLflow server where the file or directory is stored.
Model Management	mlflow.sklearn.log_model(model, artifact_path)	Record Scikit-learn models.
		mlflow.pyfunc.log_model(artifact_path, python_model)	Record a custom Python model.
		mlflow.register_model(model_uri, name)	Register the model to the model registry. The model registry is the model management and version control feature provided by MLflow, which facilitates the sharing, deployment and management of models.
Model deployment	mlflow.pyfunc.serve(model_uri)	Deploy the model as a REST API service. It is used to start an HTTP server locally to provide prediction services for registered MLflow models. After the server is started, prediction can be made by sending data through HTTP Post requests. model_uri: URI pointing to the registered model, which can be the URI in the model registry or the path of the recorded model.

tencent cloud

Sign Up

Log in

Compute

Microservice

Data Migration

Database SaaS Tool

Data Security

Application Security

Big Data

Image Creation

Internet of Things

Stream Services

Cloud Real-time Rendering

Management and Audit Tools

Edge Computing

Serverless

Relational Database

Networking

Business Security

Domains & Websites

Face Recognition

AI Platform Service

Middleware

Media On-Demand

Game Services

Developer Tools

Container

Essential Storage Service

Enterprise Distributed DBMS

CDN and Acceleration

Security Services

Enterprise Applications

Voice Technology

Natural Language Processing

Communication

Media Process Services

Education Sevices

Monitor and Operation

Distributed cloud

Data Process and Analysis

NoSQL Database

Network Security

Cloud Security

Office Collaboration

Tencent Big Model

Optical Character Recognition

Interactive Video Services

Media SDK

Cloud Resource Management

More

Magic Syntax

Jupyter Notebook Native Magic Commands

WeData Notebook Special Magic Commands

DLC Utilities (Dlcutils)

Data Utility (Dlcutils.Notebook)

Notebook Utility (Dlcutils.Notebook)

Parameters Utility (Dlcutils.Params)

MLFlow Function

Notebook Parameter Usage Example

Use project parameters in notebook

Use workflow parameters or task parameters in notebook

Parameter transmission between notebooks