JSON API with App Engine and Datastore
December 31, 2017
appengine
datastore
flask
If you are just starting with App Engine on Google Cloud Platform or evaluating it as PAAS chances are that you probably won’t have much traffic in the beginning. You will probably also need some kind of storage. If you need to store your data you might as well go with Google’s default storage suggestion for App Engine - Google Cloud Datastore.
Lets code up a simple JSON API using Flask and datastore as a storage option.
Very Brief Intro to Cloud Datastore
The Datastore is Google’s NoSQL database storage. It’s schemaless meaning you can store about any object structure including complex nested properties in form of other objects. Most of the properties are indexed by default which allows you to use them in your queries.
When using Datastore with App Engine Standard environment with Python you have to use NDB Client library. It comes with integrated automatic caching using App Engine’s Memcache which is pretty nice since it will save you some of your read quota and speed up some common queries if you have a heavy read application.
The Code
Most of the boilerplate code is the same as in previous flask tutorial which you can find here.
The JSON API we will create is a headless blog engine where users can create, update, read and delete blog posts.
Lets start with our model. It’s a simple model just to demonstrate how you would use datastore objects in an app.
class Post(ndb.Model):
author = ndb.StringProperty()
title = ndb.StringProperty(indexed=False)
# slug is a computed property and is stored everytime the entity
# is created or updated
slug = ndb.ComputedProperty(lambda self: slugify(self.title))
body = ndb.TextProperty()
tags = ndb.StringProperty(repeated=True)
# set value on create only
created_at = ndb.DateTimeProperty(auto_now_add=True)
# update value every time the entity is updated
updated_at = ndb.DateTimeProperty(auto_now=True)
We need to serialize datastore objects to JSON. Once again we turn to the trusty Marshmallow lib.
class PostSerializer(Schema):
# return url-encoded version of id
id = fields.Function(lambda obj: obj.key.urlsafe())
class Meta:
fields = (
"id", "author", "title", "slug",
"body", "tags", "created_at", "updated_at"
)
Here is the requirements.txt file.
Flask==0.12.2
werkzeug==0.12.2
marshmallow
python-slugify==1.2.4
Finally, here is our main.py app file.
from flask import Flask, request, jsonify, abort
from google.appengine.ext import ndb
from marshmallow import Schema, fields
from slugify import slugify
import werkzeug
import logging
app = Flask(__name__)
app.debug = True
# Our Datastore model
class Post(ndb.Model):
author = ndb.StringProperty()
title = ndb.StringProperty(indexed=False)
# slug is a computed property and is stored everytime the entity
# is created or updated
slug = ndb.ComputedProperty(lambda self: slugify(self.title))
body = ndb.TextProperty()
tags = ndb.StringProperty(repeated=True)
# set value on create only
created_at = ndb.DateTimeProperty(auto_now_add=True)
# update value every time the entity is updated
updated_at = ndb.DateTimeProperty(auto_now=True)
# Marshmallow schema needed for JSON serialization
class PostSerializer(Schema):
# return url-encoded version of id
id = fields.Function(lambda obj: obj.key.urlsafe())
class Meta:
fields = (
"id", "author", "title", "slug",
"body", "tags", "created_at", "updated_at"
)
post_schema = PostSerializer()
posts_schema = PostSerializer(many=True)
@app.route('/')
def index():
return "json api with flask and datastore"
@app.route('/posts.create', methods = ['POST'])
def create():
# allow only json
if not request.is_json:
abort(400, "json only please")
post = Post(
author=request.json['author'],
title=request.json['title'],
body=request.json['body'],
tags=request.json['tags'])
# write to datastore
post.put()
# return as json
return jsonify(post_schema.dump(post).data)
@app.route('/posts.fetch')
def fetch():
""" fetch all posts """
posts = Post.query().fetch()
# return as json
return jsonify({ 'items': posts_schema.dump(posts).data })
@app.route('/posts.get')
def get():
""" get individual post """
post_id = request.args.get('id')
if post_id == None:
return abort(400, "please provide a post id")
# get post by id
try:
key = ndb.Key(urlsafe=post_id)
post = key.get()
if post == None:
return not_found("post was not found")
# return as json
return jsonify(post_schema.dump(post).data)
except Exception, e:
return abort(500, e)
@app.route('/posts.delete', methods = ['POST'])
def delete():
""" delete a post """
try:
key = ndb.Key(urlsafe=request.json['id'])
key.delete()
return jsonify({'status': 'ok'}), 200
except Exception, e:
return abort(500, e)
@app.route('/posts.update', methods = ['POST'])
def update():
""" update a post """
# allow only json
if not request.is_json:
abort(400, "json only please")
try:
key = ndb.Key(urlsafe=request.json['id'])
post = key.get()
# if no entity with given id exists
# return a 404 not found
if post == None:
return not_found("post was not found")
json = request.json or {}
if 'author' in json:
post.author = json['author']
if 'title' in json:
post.title = json['title']
if 'body' in json:
post.author = json['body']
if 'tags' in json:
post.tags = json['tags']
post.put()
return jsonify(post_schema.dump(post).data)
except Exception, e:
return abort(400, e)
def not_found(message='resource was not found'):
return jsonify({
'http_status': 404,
'code': 'not_found',
'message': message
}), 404
@app.errorhandler(werkzeug.exceptions.BadRequest)
def bad_request(e):
return jsonify({
'http_status': 400,
'code': 'bad_request',
'message': '{}'.format(e)
}), 400
@app.errorhandler(werkzeug.exceptions.InternalServerError)
def application_error(e):
logging.exception(e)
return jsonify({
'http_status': 500,
'code': 'internal_server_error',
'message': '{}'.format(e.description)
}), 500
Test the API
Your app engine dev environment comes with the datastore emulator out of the box and you can view your data if you visit admin server at http://localhost:8000 once the dev server is up and running.
$ dev_appserver.py --log_level debug .
Lets shoot some curls at our API. Lets create a post.
$ curl -X POST -H 'Content-Type: application/json' -d \
'{
"author": "bob",
"title": "Master Google Datastore",
"body": "Should not be that hard",
"tags": ["gae", "flask", "datastore", "gcp"]
}' http://localhost:8080/posts.create
Response should look something like this.
{
"author": "bob",
"body": "Should not be that hard",
"created_at": "2017-12-31T09:47:00.539361+00:00",
"id": "aghkZXZ-Tm9uZXIRCxIEUG9zdBiAgICAgICACgw",
"slug": "master-google-datastore",
"tags": [
"gae",
"flask",
"datastore",
"gcp"
],
"title": "Master Google Datastore",
"updated_at": "2017-12-31T09:47:00.542255+00:00"
}
Nice! It works! You can try some other endpoints by yourself.
- Get by id
GET http://localhost:8080/posts.get?id={post_id} - Fetch all
GET http://localhost:8080/posts.fetch - Delete post
POST http://localhost:8080/posts.delete - Update post
POST http://localhost:8080/posts.update
Conclusion
Datastore is something you get out of the box if you use App Engine. Your local
development environment comes with a datastore emulator (NOTE: you might have to
install it separately with gcloud components install
cloud-datastore-emulator). Although you must invest some time to learn the
datastore it might be well worth it if you are just getting started. By using it
you can concentrate on you business logic and let the Google take care of
infrastructure. It might also save you some money if your app doesn’t have that
much traffic yet and you don’t want to pay for a separate Cloud SQL instance.