Automating Python with Google Cloud Functions

In this tutorial, we'll see how to create, deploy, invoke, and schedule a Python script using Google Cloud Functions. More importantly, I'll show you my workflow for iteratively developing and debugging this kind of system.

Create a Pub/Sub-triggered cloud function¶

Let's start with a really simple goal; to deploy and run a Python script that prints "Greetings Earthlings".

Step 1: Create a new Google Cloud Project¶

Create a new Google Cloud Project ->

Still lost? Follow the docs here ->

Step 2: Create a Pub/Sub-triggered cloud function in the console¶

Once you've set up a poject, creating your first cloud function in the Google Cloud Console is relatively straightforward. Try following the docs. Use Pub/Sub as the trigger. Then see my steps for this below.

Here's a quick video of a cloud function being created in the console.

1. Navigate to the Cloud Functions page, then click CREATE FUNCTION.

   The first time you do this, you'll need to enable some Google Cloud APIs. If you run into trouble here, it probably means you don't have the correct IAM permissions to enable services.

2. Give your function a name. Then set the trigger type to Cloud Pub/Sub.

   HTTP vs Pub/Sub

   I recommend starting out with Pub/Sub triggered functions until you need a feature supported by HTTP.

   	HTTP	Pub/Sub
   Max Runtime	60 minutes	9 minutes
   Invoke-ability	hard	easy
   Returns a response?	yes	no

   After choosing Pub/Sub as the trigger type, you'll need to select or create a Pub/Sub "topic" that your cloud function subscribes to.

   For those unfamiliar, Pub/Sub is Google's mechanism for sending messages between Google services. It literally stands for "Publish/Subscribe". In our case, the cloud function subscribes to all messages with the topic "MyTopic". That's its trigger to run.

   We haven't worked out how to publish a message yet, but we'll see that in a bit.

3. Click NEXT. If if you're prompted to enable more APIs, click ENABLE.

4. Change the Runtime to Python 3.12 or whichever version you prefer.

   Warning

   When you change the default Runtime, Google gives you some nice boilerplate files and code but if you modify the code and then change the runtime, Google will not overwrite your edited code. So be sure to change the Runtime before you edit any code.

   The default Python code includes two files: main.py and requirements.txt.

   import base64
   import functions_framework
    
   # Triggered from a message on a Cloud Pub/Sub topic.
   @functions_framework.cloud_event
   def hello_pubsub(cloud_event):
       # Print out the data from Pub/Sub, to prove that it worked
       print(base64.b64decode(cloud_event.data["message"]["data"]))
    

   functions-framework==3.*

   Every (Python) cloud function must include these two files.

   • requirements.txt tells Google which packages to install.
   • main.py contains the entry point function or class that Google runs when the Cloud Function is triggered.

   By default, the entry point is hello_pubsub. So, Google expects to find a function named hello_pubsub() somewhere in main.py.

   Furthermore, the @functions_framework.cloud_event function decorator is used to modify the hello_pubsub() funcion to properly accept and handle a cloud_event as input.

   The cloud_event parameter receives the details of the Pub/Sub message that invoked the function.

5. (Optionally) Click TEST FUNCTION to test that your function works. (This is notoriously slow, and I've gotten in the habit of skipping this step.)

6. Click DEPLOY

Step 3: Manually invoke the cloud function¶

Because we created a Pub/Sub-triggered function, we can invoke our cloud function by publishing a message to the same topic that our function is subscribed. Assuming you're starting from the Cloud Function Details page, the easiest way to do this is...

1. Click the TRIGGER tab
2. Click the link under Receive events from
3. Click the MESSAGES tab
4. Click PUBLISH MESSAGE
5. Leave the defaults in place and give your message a body like "Test 1"
6. Click PUBLISH

Navigate back to your cloud function details page and view the LOGS tab. You should see a log that includes "Test 1".

Step 4: Modify the function source code¶

We want our function to print "Greetings Earthlings". Go to the function details page and modify the source code in main.py accordingly.

Your code should look something like this 👇

import base64
import functions_framework
 
@functions_framework.cloud_event
def myfunc(cloud_event):
    print("Greetings Earthlings")

Make sure the name of your function matches the entry point name.

Click SAVE AND REDEPLOY. Then test your code by publishing a message just as you did in step 3.

Step 5: Schedule function to run every hour¶

You can use Cloud Scheduler to trigger your function to run on a predetermined schedule. (E.g. every hour, every day, the last day of every month, ...)

1. Go to console.cloud.google.com/cloudscheduler

2. Click CREATE JOB or SCHEDULE A JOB

3. Give the job a name and choose a region. (Pick the same region as your cloud function)

4. Enter a Frequency in the form of a cron expression. For example

   # Every minute
   * * * * *
    
   # Every hour, at minute 0 (0:00, 1:00, 2:00, 3:00, ...)
   0 * * * *
    
   # Every day at hour and minute 0 (today @ 0:00, tomorrow at 0:00, ...)
   0 0 * * *

   If you're relatively uncomfortable writing cron expressions (like most of us), check out Crontab.guru.

5. Pick a timezone

6. Set the target type as Pub/Sub (since we created a Pub/Sub-triggered function) and choose the same Topic that your function is subscribed to. Also provide a message body. (Our function doesn't use the message body, but it could be useful in some cases.)

7. Click CONTINUE and leave the default settings in place.

8. Click CREATE

At this point, your function should execute on your specified schedule. If you want to test it right away, select the job you just created and click FORCE RUN.

Create an HTTP-triggered cloud function¶

Creating an HTTP function from the console is 90% the same as creating a Pub/Sub triggered function. The obvious difference is to set the Trigger type as HTTPS.

The default Python code looks like this

import functions_framework
 
@functions_framework.http
def hello_http(request):
    """HTTP Cloud Function.
    Args:
        request (flask.Request): The request object.
        <https://flask.palletsprojects.com/en/1.1.x/api/#incoming-request-data>
    Returns:
        The response text, or any set of values that can be turned into a
        Response object using `make_response`
        <https://flask.palletsprojects.com/en/1.1.x/api/#flask.make_response>.
    """
    request_json = request.get_json(silent=True)
    request_args = request.args
 
    if request_json and 'name' in request_json:
        name = request_json['name']
    elif request_args and 'name' in request_args:
        name = request_args['name']
    else:
        name = 'World'
    return 'Hello {}!'.format(name)

As indicated by the code, a Python HTTP-triggered function recieives a Flask request object as input. This object includes things like the request method (GET / POST/ PUT), headers, cookies, query parameters, etc.

For our simple task, we can simplify the boilerplate code to this 👇

import functions_framework
 
@functions_framework.http
def greetings(request):
    print("Greetings Earthlings")

In this example we don't return anything, but we could. That's one of the advantages of HTTP functions over Pub/Sub functions.

Invoking an HTTP-triggered function¶

You can use cURL to invoke your function.

1. Open the TESTING tab in your function page. It should look like this.

2. Near the bottom is a "CLI test command". Copy and run it in your local terminal.

   

curl -m 70 -X POST https://us-central1-scipress-cloud-funcs.cloudfunctions.net/my-http-function \
-H "Authorization: bearer $(gcloud auth print-identity-token)" \
-H "Content-Type: application/json" \
-d '{"name": "Hello World"}'

Developing cloud functions from a local development environment¶

Listen up! You DO NOT want to be creating and managing your cloud functions from the console forever. The web interface is great for learning and testing toy examples, but any real-world project will have multiple source code files and require lots of tinkering and tweaking. You don't want to manage that process from the console.

A better workflow is to use an IDE like Visual Studio Code to write your cloud functions. Then you can deploy them from your local machine using the gcloud CLI (Command Line Interface).

How do I deploy a function from my local development environment?¶

See gcloud deploy. Google has a nice writeup about deploying cloud functions with the gcloud CLI here ->

1. Set up a project directory structure akin to this

   my-project/
     funtions/
       main.py
       requirements.txt

   If your main.py imports code from another local python script, you should include that script in the functions/ directory.

   my-project/
     funtions/
       main.py
       utils.py
       requirements.txt

2. In your local terminal, run gcloud functions deploy (NAME) with your desired parameters. For example,

   gcloud functions deploy MyFunctionName
   --project=my-project-id
   --region=us-east1 \
   --runtime=python312 \
   --source=functions \
   --entry-point=myfunc \
   --trigger-http \
   --no-allow-unauthenticated

   In this example, source=functions tells Google to upload the functions/ directory.

   If you save this command to a file like deploy.sh, you can just run ./deploy.sh in your terminal to execute the command.

Notable parameters¶

• Use --timeout to change the default run time limit. E.g. --timeout=3600 sets the max runtime to 3600 seconds (1 hour).
• Use --memory to change the default memory limit. E.g. --memory=2Gi sets the memory limit to 2 Gigabytes.
• Use --ignore-file to prevent a specific file from being uploaded. E.g. --ignore-file=deploy.sh prevents functions/deploy.sh from being uploaded. (Alternatively, use a .gcloudignore file.)

How do I deploy multiple functions that use the same code?¶

Suppose you have Python functions foo() and bar(), and you want to create a cloud function for each. However, each function imports a helper function baz() from the same utils.py file. How do you set up a local project where you can deploy each function without duplicating code?

Set up a directory structure like this

my-project/
  funtions/
    main.py
    utils.py
    foo.py
    bar.py
    requirements.txt

• foo.py defines foo
• bar.py defines bar
• main.py imports foo and bar

def foo():
    print("foo")

def bar():
    print("bar")

Then run a deploy command with the parameter --entry-point=foo to create the first cloud function and another deploy command with the parameter --entry-point=bar to create the second cloud function.

How do I schedule an HTTP function to run on a recurring basis?¶

You can use Google Cloud Scheduler to schedule an HTTP function just as you can with a Pub/Sub triggered function. However, HTTP functions require a bit of extra work.

1. Create a service account with the Cloud Run Invoker role
2. Set up a Cloud Scheduler job. While creating the job, choose Auth header = OIDC token and select the service account you just created.