GitHub Integration with MongoDB Stitch

Greetings, fellow travelers!

I wanted to take some time today to talk about MongoDB Stitch, a new back-end-as-a-service offering by MongoDB, and walk through making a simple web page to view GitHub Issues.

If you are unfamiliar with MongoDB or other Document Store databases, I highly recommend you take some time to read more about the differences between the different database types,  go through MongoDB’s Getting Started guide, or even take one of the awesome courses offered via MongoDB University.

Fortunately for us, since MongoDB Stitch is a back-end-as-a-service, we don’t need to be experts in setting up and administrating a MongoDB database; they’ll do it for us!

Note: MongoDB Stitch is still in beta. Some of these features I describe may change before their 1.0 release.

We’ll need a few things:

  1. A MongoDB Stitch Account
  2. A MongoDB Atlas Account (creating a Stitch Account will create an Atlas Account)
  3. The MongoDB cli (optional)
  4. A GitHub Account
  5. A GitHub Repository you manage and can create webhooks on
  6. Your Favorite Text Editor

The Goal

We’re going to create a web page that displays a list of created issues in a set of GitHub repositories. When an issue is created in GitHub, it will fire a webhook to our Stitch application, which will massage the data some before storing it in the Atlas database. Finally, the web page will access the Stitch Atlas service to list the issues.

Let’s get started.

Create a Stitch and Atlas Account

Getting started with Stitch is simple. Head over to their landing page and click on the “Get Started Free” option. This will walk you through creating a Stitch and an accompanying Atlas Account and Cluster. If you have any questions on this stage I highly recommend you look through their documentation on getting started with Atlas, and creating a cluster. Their explanation is concise and easy to follow.

Create Your Stitch Application

Getting started with a new Stitch Application is as simple as logging into the portal, clicking “Create New Application,” and giving it a name and cluster! For this, I will be creating a Stitch Application called GithubIssueList and assigning it to my AdvPlatCluster Atlas cluster.

Note: if you just created this cluster, you may have to wait a few minutes for it to show up in the list, as the cluster may still be constructing itself.

1_newStitchApp

Authentication and an API Key

Once complete, you’ll be taken to your App’s Getting Started page. The first thing we will need to do is turn on Authentication. Anonymous Authentication is a great way to get started, but not a great way to build a secure application. Let’s go to the sidebar under “Control” and navigate to the “Authentication” page.

2_controlAuthentication

We’ll enable API Keys for authentication and make a new API Key: GHApiKey.

4_noteUserID

Take note of the API Key and the User’s ID field. We’ll be using them both later.

Set up the Database

Now we’ll need a place in our cluster to store the data. Let’s navigate to the Atlas Cluster’s section in the sidebar, click on the cluster we linked earlier, and then on the “Rules” tab. You’ll see there is already a collection created for you of format ​app-{someNumbers}.items. We’ll create a new one for the database ​ghdata and the collection issues6_newCOllection

When we receive our webhook data from GitHub, this is the database and collection we’ll put the documents into.

You’ll notice that we have some preconfigured permissions. By default, both read and write permissions are allowed if the document’s owner_id field is equivalent to %%user.id. We should know that %%user is a special expansion that can be used in Stitch Rules and Pipelines. For our purposes, when querying the database via Stitch while authenticated with our API Key, we will only be able to read or write documents whose owner_id field is equal to the User ID field we wrote down earlier.

You’ll notice there is also a Filter tab. This, too, comes preconfigured:

7_filters

The default filter says “When true, only show documents whose owner_id field is equal to the current User’s ID field”.

We should note that filters are applied before rules are.  This means that even if we remove the preconfigured Read and Write rules around owner_id, we would still be unable to read another User’s documents as the Filter would exclude them before the Rules could be evaluated.

At this point you might want to connect to your Atlas cluster to ensure you have solid connectivity and your IP address is whitelisted.

Set up GitHub Integration

With our User Created and our Database up and running, we need to allow our Stitch application to receive data from GitHub. In the sidebar under Services, click “Add a Service”, choose GitHub and give it a name (I am calling mine IssueService).

8_addService

With our IssueService created, we will add an Incoming Webhook to it. Give the Incoming Webhook a name, and a secret.

Be sure to take note of the secret as we will need it when we configure the GitHub repository to send us information.9_nameAndSecret

 

 

 

The Webhook Pipeline

Once again, Stitch has set up a nice set of default stages in the pipeline. Pipelines are sequences of actions defined in JSON. Taken from their documentation:

  • Pipelines are written in simple JSON.
  • Pipelines can be constructed in the UI and referenced by name in code, or coded directly with SDKs.
  • Pipelines can use expansions to pass information from one stage to another or incorporate information from outside the pipeline stage
    • Information can be passed from one stage to another with let.
    • Information about requesting user can be incorporated with %%user.
    • Global variables can be defined in Stitch and referenced with %%values.
    • Information about the current or prior state of a document or field can be incorporated through MongoDB expansions.
    • Some services like Twilio can execute a pipeline as a response to an incoming webhook.

The default pipeline for the GitHub incoming webhook has two stages: a literal stage, which binds data coming in from the webhook’s body, and a match stage, which takes the data passed to it from the literal stage and filters it based on an expression. This match stage checks to see if the property “pull request” exists in the document and if the value stored in the property “action” is “opened” or “synchronize”.

For example, this document would pass the match expression:

But this one would not:

While this is great for pull requests, we are interested in tracking issues, so let’s edit the default mapping action to the following:

This states that we only want to match documents who have a property called “issue” and whose “action” property is “opened”. If we allow “synchronize” events, we may wind up with duplicate data in our database.

With that, our pipeline is ready to receive valid GitHub issue open webhook events, but aside from filtering the data, our pipeline isn’t doing much with it. Let’s look at the Issue Event payload from GitHub. There is a lot of data in there, and while we have the 25 GB Free tier of hosting, it is wasteful to store all that extra data we won’t be using.

Projecting the Data

To filter down some of the unused data, let’s add a new stage to our pipeline, and change its action to “project“. We’ll use this projection action to include only the fields in our document we want. In particular, we will include:

  • issue.html_url
  • issue.id
  • issue.title
  • issue.comments
  • issue.created_at
  • repository.name
  • repository.html_url

This will cut down our data usage significantly.

We do so by defining the following in the event:

At this point our pipeline should look like this:

10_pipelineStages

We accept the documents, filter them, and then we map the unnecessary fields away. There are two last steps we need to complete:

  • Give our documents an Owner
  • Insert the documents into our database

Give the Data an Owner

Adding an owner_id field to our data requires a new stage in our pipeline. Let’s create a new stage whose action is “expr” or expression. Expression stages allow documents to be filtered and then modified before being passed on to the next stage. We’ll first need to enable the “Bind to %%vars” toggle in our stage and insert

as the content. Next, we will set the main body of the expression to be

where <API USER ID HERE> is replaced by the value of the ID field for the Api User we created earlier.4_noteUserID This returns a whole new document with two top level fields owner_id and issue, with the value of the issue field set to the output of our projection.

Insert the Data into the Database

The final stage we need to add is the easiest! Hit “Add Stage” one more time and

  • Change the “Service” drop down from built-in to mongodb-atlas
  • Change the Action from find to insert
  • Modify the Database value to  ghdata
  • Modify the Collection value to issues

In the end it should look like this:

11_insert

This will connect to our Atlas service and insert the document into the issues collection of the ghdata database.

And we are done! Hit create and your pipeline should look like this and be ready to accept incoming webhooks:

12_resultPipeline

 

Configuring GitHub

Before we jump over to GitHub, you should notice that creating your incoming webhook provided you a webhook url:

13_webhookUrl

Copy it to your clipboard, and head over to a GitHub repository you want to enable this webhook with. Click on Settings, then Webhooks and the “Add webhook” button.

14_addWebhook

Paste your webhook url into the Payload Url field, change the Content Type drop down to application/json, and set your secret to the secret you configured in your Stitch Application.

15_configureHook

Next, choose “Let me select individual events”, uncheck the “Pull Request” option, and check the “Issues” option.

16_onlyIssues

Finally, hit “Add Webhook” to enable it!

If you click on the webhook and scroll to the bottom of the page, you can see a “Recent Deliveries” section. You can click on each of those identifiers to see if our pipeline returned an error code and what the request looked like.

17_recentPayloads

Making a new Issue

At this point we have our Stitch Application up and running, ready to receive data, and our GitHub repository is ready to push it on new events! Let’s open a new GitHub issue on our repository to see what happens18_firstIssue

The recent deliveries shows a success

19_issueDelivered

And the Logs console in our Stitch Application show success as well!

20_stitchDataLogs

Right now if you were to connect to your Atlas cluster use the ghdata db and run db.issues.find().pretty() you would see something like:

21_prettyDataGotIn

Our data made it all the way through!

Writing the Web Page

The lion’s share of our journey is over. All we need is to write a web page to connect to the Stitch Service and view the data. You can use any framework you like as there are Stitch clients for Node, Javascript, Java, and Swift, but I’m going to write a simple static HTML page suitable for a custom “New Tab” page in your browser. If you want to skip to the end, you can check out the final code in my GitHub repository.

Our page will have a simple title and a div placeholder where we will insert the list of issues we get back from our Stitch Application.

Outline

I’ve downloaded the Bootstrap css files as well as the Stitch driver so I won’t need online access to them (since this will be a “New Tab” page). I’m also including a config.js file in which I’ll define some constant variables to use.

The config.js is there to expose a number of const variables to use when logging in.

The APP_DB and APP_COLLECTION variables come from the database and collection we are storing our issues in.

APP_ID is the unique identifier of your Stitch App. You can find it on the Stitch console in the “Clients” sidebar.

23_appID

APP_API_KEY is the API Key the client will use to authenticate with the Stitch Service. You can get the key by going to the Authorization page, editing the API Keys authorization provider, and hitting “Show Key” on the API Key you wish to use.

22_authProvider

Finally, we can write the Javascript client code:

In these two functions we do the following:

  1. New up a StitchClient connected to our Stitch Application
  2. Request a connection to a MongoDB Atlas service and get the database
  3. Authenticate using our API Key
  4. Query the issues collection for all documents (but limiting it to 10)
  5. Log the documents (because debugging is fun!)
  6. Write some html for each document we received
  7. Set the inner content of the div named “issues” to our generated html

This should give us:

23_theResult

We did it!

So What Have We Accomplished?

We’ve managed to accomplish a lot, if you’ve been following along:

  1. Set up a new MongoDB Database hosted in the cloud via Atlas
  2. Created a Stitch Application
  3. Secured the Application via API Keys, database rules, and filters
  4. Set the Application to respond to incoming webhook events by updating a database
  5. Configured a GitHub repository to provide events when an issue is created
  6. Wrote a Web page to connect to our Stitch Application and display the data in it.

Where Do We Go from Here?

  • Spruce up the Web Page UI
  • Write the Web Page in a full framework like MVC, Spring,or Angular
  • Write Another WebHook for Issues being deleted to remove them from the database
  • Write a mobile App to view the Issues using the Java or Swift Clients
  • Connect the GitHub Pipeline you made to another Pipeline (like Twilio or Slack)
  • Build your own App!

Stitch is a powerful and flexible back-end-as-a-service. As it grows out of beta and connects to more services and pipelines, I can’t wait to see what we can build!

Happy coding!

My Introduction to Flutter

I sat down last weekend and spent some time with Flutter. Though I only wrote one small sample app in it, and I’m not a mobile developer by trade, I wanted to write down a few of my thoughts on it.

The Good

Flutter is Written in Dart

I’ve become a big fan of Dart over the past few weeks. It lives in a space that is approachable from both a C# and a JavaScript perspective, has an approachable package management system, supports functions as first-class objects,  and approaches “strong typing” via dynamic objects with type annotations.

Flutter Compiles to Native

The title says it all. Apps written in Flutter compile to native code on iOS and Android.

Flutter Supports Hot Reloading

This is something that you can see in demos and videos, but it has to be experienced to be fully appreciated. It cuts down on development cycles significantly, and is just plain satisfying to hit the “r” button and watch your changes spring to life without bringing the application down and up again.

The Bad

Flutter is in Alpha

Depending on how adverse you are to deploying alpha software to production, this could be a good or a bad thing. The bright side to Flutter being in alpha means it can only get better!

Some of the More Useful Examples are Buried

I know that this is less about the framework and more about the support system, and that opinion on this may vary from person to person, but the Getting Started tutorials for Flutter, despite being very good at explaining the framework and how to get up and running, do not, in my opinion, dive into the proper way of building multi-screen apps early enough.

Consider the sidebar:

navigationBar

The section that covers making multi-screen apps, and introduces some of the core concepts is not until “Routing and Navigation”, and the previous examples, while good and useful (Building Beautiful UIs with Flutter was particularly helpful), give examples with no Navigation. I highly recommend developers new to Flutter go through the “Get Started”  and “Build UIs” section, but before they sit down and write any code, they check the excellent examples on the Flutter GitHub Page.

A Final Important Thought

This is what I wanted to really talk about:

There is no Template Engine Or WSYWIG Editor

When I first started developing with Flutter I was somewhat surprised to learn that they do not have WSYWIG editor like other UI frameworks (XCode and iOS,  Micoroft Blend and WPF, Android Studio and Android). If you jump in the Flutter Gitter, the maintainers have mentioned that they don’t have any public plans to release one:

noPlans

And you know what? I’m okay with that.

Earlier, I mentioned there is a magic to hot reloading your app, and this is why I don’t feel like Flutter strictly needs a WSYWIG designer. With Flutter, the app is the designer. You don’t like the style of a particular word box? Go find the Widget that contains it, edit the style property, hot reload the app and marvel at your design acumen!

The biggest downside I see is it raises a barrier to entry for people who don’t know how to code and want to design an app’s user experience.

Is there a space for a WSYWIG designer? Of course!

Is Flutter a great framework without one? Definitely!

Will I be using it for my future mobile endeavors? You bet!

Happy Coding!

Flutter on Windows without Android Studio

tldr; Install Gradle

Lately I’ve been falling more in love with Dart and Go, two modern open-source languages by Google, and with the recent murmuring around the Magenta kernel and Fuchsia OS,  I’ve been spending more and more time working in both languages (even writing an Epub reader in Dart).

While I’ve been enjoying writing console apps and libraries, I wanted to try my hand at writing some U.I. apps.

Enter Flutter.

Flutter “is a new mobile app SDK to help developers and designers build modern mobile apps for iOS and Android.” and it feels like a lighter-weight competitor to Xamarin. It has a great getting started guide, so I began with the setup.

Note: I’m running Windows 10 Creator’s edition with Hyper V disabled and Xamarin Installed. Your mileage may vary

After git clone -ing the repo and adding the flutter/bin folder to my PATH, I ran the flutter doctor command and got the following:1_flutterDoctorNoSdk

Flutter couldn’t discover my Android SDK (which I had installed via Xamarin previously), which was no problem: I simply set my ANDROID_HOME environment variable and it picked it up.3_sdkInstalled

Android Studio not being installed was problematic for two reasons:

  1. I was on a bad coffee shop WiFi and probably couldn’t download the entire 1.5 GB installer in a reasonable amount of time.
  2. I am a big fan of the Dart community, in particular Danny Tuppeny’s (@dantup)  Dart Code Visual Studio Code extension which makes developing Dart libraries a breeze, so I’d rather use VSCode and his extension over Android Studio.

With those considerations in mind, I decided to skip installing Android Studio and just run

flutter create myapp

4_flutterCreateWorksNoAndroidStudio

Which made a perfectly good Flutter application I can open and work in with Visual Studio Code.5_aWellCreatedFlutterApp

So let’s flutter run !6_cannnotLocateGradeInstallAS

Unable to locate gradle. Please install Android Studio

So that’s what Flutter needed from Android Studio! At this point my download of Android Studio was 50% complete with another hour to go, so I decided to download and install Gradle manually, update my PATH environment variable and give flutter run another try:8_gradleWorksButBadVM

I’m getting an entirely different error now:

Could not reserve enough space for 1572864KB object heap

A quick Google of this instructed me to update my gradle.properties file with

org.gradle.jvmargs=-XX\:MaxHeapSize\=256m -Xmx256m

Now flutter run took me further, informing me that I had not accepted the license agreement for Android SDK Build-Tools 25.0.3. Which was actually somewhat misleading. In fact I had not even installed the 25.0.3 Build-Tools.

A quick trip to the SDK Manager to install and accept licence for the 25.0.3 Build-Tools, and one last flutter run got me to a successfully running Flutter app, all before Android Studio finished downloading.17_runningProperly

18_runningInEmulator

Success!

From here you can iterate and improve on their sample apps or get started with your own!

Happy coding!