Google released a set of community mobility reports using their anonymized movement data from Google Maps:

Officials could use these to compare policies and communication tactics across similar jurisdictions. I do wish the data were more granular, but I understand the privacy implications that would pose.

Shaming South Dakota

Let's compare Sioux Falls South Dakota with Louisville, Kentucky – two similar cities.

As of March 31, South Dakota is the only state not implementing social distancing or shelter-in-place measures. Kentucky is.

Here is the mobility report for Minnehaha County that contains Sioux Falls. Sioux Falls is South Dakota's most populous city.

Compare that with Jefferson County, containing Louisville, KY:

There's a more substantial drop in retail and workplace attendance in Louisville in Sioux Falls. There is also a higher increase in park attendance.

There's a lot of nuances, though. For example, Sioux Falls has COVID19 guidance their site. But without a statewide order, it may not be striking a lot of the population as necessary.

Californians hate parks?

I also found this interesting. Louisville and Sioux Falls had a significant spike in park attendance. San Diego County had a significant drop.

It's possible San Diegans hate parks. Or – more likely – the end of winter in Kentucky and South Dakota was encouraging people to go outside.

Downloading the Reports

To download these, head over to the Community Mobility Reports page, and find your state.

The first page of the report shows aggregated metrics for the state, with an explanation of what they are:

As you scroll down, you can get individual county reports.

If you end up finding them useful, please let me know how by tweeting me @MrMaksimize.

Successfully applying data and technology in government is hard work. Telling a good story about these applications is just as challenging.

“What makes a good data story?” does not interest me.

“What makes a successful data project?” interests me. “How do successful data teams operate?” interests me. “How do we perpetually sustain good government?” interests me.

A good data story

A good story doesn’t care about the tech or the tools. It cares about the people that “led the effort”; The politician that got credit for it; Or the organization that wrote about it.

A good data story has just the right amount of buzzwords. It will fit into a 1,000 word blog post (like this one), Regardless of the underlying project’s value or effectiveness.

A good data story is about using machine learning, In the form of advanced neural networks, Across data gathered by a $5 million innovative sensor array, That stores the data in a Hadoop cluster, To determine how many drones just flew by.

Yet, nobody asked how many drones just flew by.

A good data story chases talking points — and forgets to solve a problem that matters.

Sorry, that’s not interesting.

A good data project

A good data project identifies a problem that is worth solving. It requires that city staff across departments work together; Means building buy-in for data access and process change; And accounts for regulations that are decades old.

A good data project is about details, The right technical approach to create the right solution, And the process change that influences the right outcome.

A good data project is about the Streets engineers who spent years understanding street paving, So they can spend 20 hours explaining their processes to the data scientists, Who will to figure out how to run Python on our legacy servers, So that the front-end developers can provide a usable interface to street paving analysis, And the program manager who championed the project can implement it operationally.

A good data team

A good data team builds relationships with employees across all city disciplines; They use design thinking techniques to identify hard problems; And make disciplined technology choices to deliver the right solution.

They save countless hours for Streets engineers, By helping them them auto-generate reports.

They analyze historical data and inform dispatch configurations for Fire/EMS, Helping doctors make the case for adjusting dispatch plans.

They evaluate fleet vehicle GPS movement and propose new routing patterns for delivery trucks, Allowing for more efficient movement of materials between city offices.

They build Police dispatchers a real-time dashboard with emergency call hold times, By installing a Firefox plugin.

A good government

The aforementioned projects are impactful; Cost almost nothing; And will likely never see a press release.

We’re okay with that.

We will tell our story to the people that benefited from our work: The street engineer who will now save time running reports. The truck driver that spends less time in traffic, completing deliveries faster. The dispatcher who will more effectively take calls, saving more lives.

We are not here to tell good data stories. We are here to do good work. We are here for good government.

We recently completed a small sprint and made a few fixes and improvements.

We love listening to our users, and one of the most requested features has been a way to type in an address and see what streets have been or will be repaired around where you live or work.

We implemented an autocompleting search using Mapzen’s awesome search api. Now you can search for a place (like Balboa Park), or an address and it will zoom right in for you:

We also made a few performance tweaks by moving hosting from Github Pages to Amazon S3 and CloudFront to align with our overall infrastructure and deployment strategy. Hopefully sometime soon we’ll be able to make a few tweaks like we did on data.sandiego.gov to significantly increase the loading speed.

Finally, there is one more tweak we made, but we’re not ready to talk about it yet, because it needs a bigger piece before it goes into production. However, you can probably find it by looking at the open source code driving StreetsSD.

As always, we learn, iterate and continuously improve our products. We Love To Hear and Implement your suggestions!!

But we love pull requests even more!

The Chief Sustainability Officer is a recent position that has been popping up as well. The CSO works to plan for upcoming challenges brought on by a changing climate. She also works on reducing the City’s greenhouse gas emissions and carbon footprint.

Both roles involve dealing with City and regional data. Cities have a trove of data they collect, store and manage, but it’s usually not managed in a strategic way. The CDO knows where the data is, what the caveats are, and how to connect to it and perform automated analysis. The CSO needs continued information on policy baselines and to evaluate policy decisions. Neither of us want to continue to receive data via email or in a PDF.

Both roles have very little definition, very little staff and almost no resources. The CSO and the CDO, have to run their teams as lean startups. Multi-million and multi-year IT projects are off the table. Both are outside-the-box thinkers and consider “that’s the way we’ve always done things” as a challenge.

At the City of San Diego, the CDO and the CSO are natural partners. The CSO has a deep understanding of policy and business objectives. Technology and data, are not always her cup of tea.

The CDO has a strong technical background. He knows where the data is, what it means, and what the caveats are. He knows how to leverage it solve business objectives and watch policy decisions. He doesn’t want to sit in meetings, trying to form policy juggling people’s egos. That’s where the CSO comes in.

Let’s take a concrete use case. As I’m sure you’ve heard, the City of San Diego is getting a deployment of Smart Street streetlights. It’s the first and largest Internet of Things deployment in a city in North America. These streetlight sensors have a lot of different capabilities (blog post coming soon). They have the potential to improve parking management. City staff can use them to monitor pedestrian and car traffic. We can use them to track our environment.

Without reading the API documentation it’s hard to understand what’s possible. It takes someone with a background in software to do that. Then, he would have to translate it to humanese. The CDO happens to enjoy reading that kind of stuff. The CSO definitely does not.

The CDO has a full understanding of the sensors’ capabilities. The CSO has in-depth knowledge of what we need to monitor and change to meet climate goals. Together, we can decide how to utilize this technology to meet her business objectives.

These sensors also need a coordinated policy effort to manage the data. The CSO happens to be good at that as policymaking is part of her job. She can help me push through the policy work that I need.

Our unique skills and expertise complement our shared drive for change and our similar desire to get sh*t done.

The data movement in cities has followed an upward curve similar to the sustainability movement. They have also followed a similar time frame. This is not by accident — both of these movements are new ways of thinking and new policy directions. This is not simple correlation. These roles are very much intertwined and are supporting and enabling each other’s day to day work.

Be honest. When you read the words data automation you get a sudden rush of melatonin to your brain, your eyelids get heavy, and you get an uncontrollable urge to fall asleep. Don’t be ashamed; you are reacting to these words in a similar manner to 99.5% of people on the planet. Bear with me though for just a few paragraphs while I try to explain why it matters, and how we do it here at the City of San Diego.

Before we get started, let me promise to not use any of the following:

• Mathematics (or any subset thereof)

• Buzzwords

• Tech Jargon

The Quick Win

One of the best case studies to make obvious the benefits of data automation is **StreetsSD. We started working on this project with the City’s Transportation and Stormwater Department (TSW). They asked us to build a map of paving projects. We got a spreadsheet of data from the department, loaded it into [our mapping tool](http://carto.com/)** and wrote some code. Boom, we had a map.

The Ugly Loss

Most web maps, charts, and data visualizations you see in newspapers, and even those created by a lot of cities, usually stop there. That’s actually OK because a lot of these things don’t need to be updated automatically. However, the goal of StreetsSD is to provide an up-to-date status of work to city employees and residents. Keeping the map updated is important.

The typical City approach in these situations is to have someone “run a report” and upload it somewhere to update the map. Most of the time, “running a report” actually means this:

• Extract data from a database, using a query (that could change, resulting in different data).

• Complete 1–40 manual steps in Excel to clean the data. These steps are usually not documented anywhere and could easily vary for each run of the report.

• Send it or upload it somewhere.

This approach inherently has several problems:

• Data is inconsistent, meaning source of truth is inconsistent

• Bad data potentially causes bugs in the application

• City employees waste a HUGE amount of time

• People might forget to actually run the report

• We can’t expect someone to run a report every day (or with any kind of frequency that a continously update data source needs).

Needless to say, this sucks. And even though we’re creating a great operational tool, we’re costing people time and risking the release of wrong information.

At this point, being the wise reader you are, you might be thinking that the outcome would definitely be worth the cost in the case of just one map. To which I’ll respond with:

This is not a problem with just one map. It would occur with any dataset we provide as open data on the portal, anything we build on top of those datasets, and any internal or external reports we regularly generate. Therefore, this is a problem that needs to be solved at a systemic level.

Redemption

Our philosophy has always been to let machines do what they do best — updating data, re-running things, keeping track of things — and humans do what they do best — making those fuzzy decisions that only our brains can.

We needed a flexible and extensible solution that could scale across the organization as a whole.

We turned to an open source project called **Airflow**. It has become the tool of choice for Spotify, IFTTT, Lyft, AirBnB and many others for solving this exact set of problems. Airflow has several advantages:

• Built in Python — so it’s extensible with any of thousands of Python open-source packages

• Modular — new connections to different data sources are easy to write

• Open source with a strong community — so there’s plenty of support and continuous updates

• Scalable — as we need to do more and more, Airflow easily scales

We called our Airflow deployment Poseidon because codenames are cool, San Diego is on the Ocean, and Poseidon rules the sea. Plus, we get to use images like these:

The basic idea of Poseidon is this:

• Get data from a source (database, spreadsheet, map, website)

• Do some stuff to it (geocode it, aggregate it, clean it)

• Upload it (put it on the cloud that is backing our portal and a variety of other applications)

• Run it on a schedule (once every 5 minutes, OR once every day, OR on every odd day of the month at 3:02 PM)

Simple, right?

We’re now doing this for all of our datasets (way harder to implement than explain). This means we’re also doing it for everything built on top of our data, such as StreetsSD, the portal, and various other visualizations.

The Open Sea

Now you’re probably thinking: whoop-de-doo, I’m a resident and I don’t know data, so I really don’t care that you now have automated data.

We know. That’s why we pushed it further.

Look back at the diagram above. Those are just dependent pieces that run in a defined cycle. What if we flipped out some of those pieces and got this?

Automated data is now starting to get interesting. Because of Poseidon’s flexibility, and the fact that it’s just a bunch of coordinated tasks, we can send automatic notifications, alerts based on thresholds, and all kinds of other cool stuff that would never have been possible without automation.

But if I told you what we were planning next, I’d ruin the surprise. You’ll just have to wait and see.

One of the reasons we wanted to run our own data portal is the flexibility we have to change it and add functionality.

Today, we’re putting the pedal to the metal on those desires. We initially launched the portal based on JKAN, but with modified schemas, layouts, and branding. Because of how fast we moved, we put off thinking about speed and performance.

Since the dust settled a bit, we had a chance to do that.

Lighthouse is a tool developed by Google to test web pages for performance, accessibility, and more. The first time we ran it against our portal, here’s what we got:

See full report

30 out of 100, with a lot of marks against the portal due to loading speed. Time to first paint is a metric that measures when the primary content of a page is visible. That number means it took 5.3 seconds after you navigated to the portal for anything to show up.

We did badly on plenty of other metrics, including SSL encryption (HTTPS) and offline browsing.

So why does this matter (besides our obsessive perfectionism)? KissMetrics has a great infographic about this:

In short, we were losing an estimated 25 percent of portal users because of the page loading speed. Not good.

My friend and former Code for America co-fellow David Leonard (who also got me into Polymer components that the portal heavily uses ) helped me analyze where the bottlenecks were in our page loading time and gave me some tips about how to speed up the portal.

We implemented https and service workers (for security and offline browsing and caching) and optimized how components load. The portal is now encrypted, and you can browse it offline.

I’m not going to go into too much detail, but you can see the pull request here. We tested again and got a score of 96:

See full report

Let me use a few buzzwords to describe what happened here, for those fond of them:

We used an agile process to iterate on the technology underlying the performance, availability, and scalability of our portal architecture to decrease page loading speed, increase cybersecurity, and improve an already great product with enhanced user experience.

Or in English:

We did a thing that most organizations do: we improved something we launched, because no one ever gets everything right the first time. We’ll do it again, and again, and again. This is just the first of many changes and enhancements we will be making.

Enjoy your [faster] portal, San Diego!

Please see the DevPost Project Page for more info about the project, the hackathon and the team.

Inspiration

Endangered species in conservation areas are threatened to the point of extinction by poachers. Local authorities are resource constrained and are not able to monitor an entire conservation area for poaching activity — particularly the interior of a preserve. Humans in general would not be very good at this either because of the high-friction way reporting would have to be done. Our solution was deploy low-cost connected sensors throughout the interior of a conservation area to monitor the area for suspicious activity.

What it Does

Savvy Savanna allows low cost IoT — the sensor and the button — to be deployed in conservation areas and surrounding villages. The system allows people to press a button to report suspicious poaching activity and solar-powered field sensors which automatically report potential poacher problems in the savanna.

Sensors in the savanna detect sounds, such as helicopter noise, that trigger an alert to be sent to the cloud and subsequently to the appropriate authorities.

The system alerts authorities of poacher activity detected by the system. The authorities receive a text message alert and a location of where to respond.

Challenges

Cellular signal reception and WiFi network coverage is limited in the savanna. The sensors need a way to transmit information to the cloud. To solve this problem, the sensors in the field are part of a mesh network. This is technology currently utilized in “smart cities” with interconnected devices. Field sensors can be networked together with only certain sensors directly connected to the internet network.

Community members may be targeted for attack by organized crime groups if they are seen as reporting poaching activities. To solve this problem, “the button” component of the system is a way that community members can anonymously report potential poaching activity in their neighborhood by simply pushing a button.

Savvy Overview Prezi

Cost Estimation

Our rough estimates are that with the current architecture, deploying one of these one per 0.2 square miles would cost around $20,000. However, as we make changes to the hardware and implement mesh networking, this cost could significantly go down.

How we Built The Sound Sensor

Hardware

The hardware consists of a Particle Electron Board. These are Arduino based boards that have a connection to the internet through a 2G network available in over 100 countries in the world.

Particle Electron

They are backed up by Particle’s cloud infrastructure. This allows us to trigger webhooks directly using the firmware on the board. We can also use this architecture to expose different functions on the board as POST requests, allowing us to build an API for our IoT device.

We started by wiring up the Particle board with: * Analog microphone * LED for monitoring the status of the run loop (debugging) * LED for monitoring the input volume of the sound coming into the mic

There is an additional LED on the electron board itself. We used that to communicate that the sound threshold has been reached an a notification is being triggered.

Particle Wire Up

What Happens

The sensor sits there and listens. Arduino devices operate on a loop, so it samples the sound volume every 2 seconds. Since it’s an analog input, the volume will be anywhere from 0 to 4096. When the volume spikes over 2000, a command is sent to Particle Cloud. This triggers a POST webhook with the data sent from Particle to AWS API Gateway which then relays the input to AWS Lambda. The Lambda callback function then triggers the Twilio API to deploy text messages.

Lastly, we use the MapBox static maps api (you can use API to generate a static map image with the location and send that through as Twilio MMS. Since we don’t have a GPS sensor on the button or the Electron just yet, we used StaticMapGen to prototype.

How we Built The AWS IoT Button

Hardware

While a sound sensor is nice to have for continous monitoring, our idea was also to have a way for community members to anonymously report poaching, avoiding retaliation from organized crime groups. We used an AWS button as a stand-in. However, since it’s only wi-fi enabled and carries no gps chip, ideally we would build a custom one.

What happens

Because of the modular architecture we already had in place it was fairly simple to add the AWS IoT trigger as an additional trigger for the Lambda function.

How it all fits together:

Overall Architecture

Code

Grab the Arduino code for the firmware

Grab the Lambda Function JS

Next Steps

We will probably continue hacking on this project, as this is an important challenge and we think we may have a way to solve it. We still have quite a bit of ways to go, specifically:

• GPS Enable the sensors

• Solar Enable the sensors

• Provide protective covering for the sensors

• Provide cloud infrastructure for data aggregation and monitoring

• Train a model to recognize sound (elephant trumpet vs helicopter noise) and deploy it to the devices on the edge. This may require switching to integrating something like the Intel Edison chip.

• Switch devices to using a low-power WAN configuration, so they can operate in a mesh with only a few points of direct network connectivity to the web.

• Implement something like Carloop.io in ranger vehicles to prevent false positives caused by ranger vehicle noise.

Keep track of what we’re up to and jump in over in the Github Issue Queue

Overall Architecture

Jekyll

Overall, StreetsSD is a Jekyll site. This allows us to avoid maintenance costs, performance issues, and complexity associated with running database-driven sites. We host it using Github Pages for the wonderful price of $0.

Jekyll Outline

Everything you see outlined in pink above is Jekyll and custom JS. We used the San Diego Style Guide for colors and layouts. The charts on the bottom, the layers, and the text in the explainer boxes are controlled by the views.yml in the repo.

All the dynamic components you see from the layers (FY14, FY15, etc) to the charts and totals are populated through custom SQL queries against data in Carto. They are constructed using the squel library in the sqlBuilder file.

As an example, when you click on the “Work in FY–2016” layer, an event is sent that constructs and sends the query to Carto to display results on a map. Based on the views.yml file, we know that layer has WorkType and WorkByMonth charts, so one more query is constructed and sent to Carto. After additional JS data processing, we are able to display the charts.

Carto

The interactive map itself is all Carto. Most of the visual components (e.g. line styles, legends, hovers) are stored in Carto as well. We don’t love that, because those should be stored in code, but we have an issue on the backlog to address it.

Carto editor

The map in Carto has two “layers” — one for OCI and one for Streetwork, both based off sdstreets_query table. Carto’s model for associating layers to datasets ties them together permanently. By having a virtual dataset that we can impose queries on, we are eliminating the tie between a dataset and a map, giving us more flexibility to flip out the underlying datasets if we need to change schemas.

Then why two layers from one table? Because OCI layers and streetwork layers have slightly different fields and labels. Carto’s interactive UI is tied to a layer, and we had to separate the two. Then, we impose the queries on the layer accordingly. We can probably condense this to one layer using more advanced conditional logic, but we’ll stay away from that for now.

Deployment

We use the master branch as the bleeding edge. That gets deployed to a staging site continuously by CircleCI. The production branch gets deployed to the gh-pages branch, which is what feeds StreetsSD. We get notifications about each step.

Datasets

cgstreetscombined

We created this dataset by merging the City’s official street file geographies with essential data from the City’s Pavement Management System. Every other set of data is joined against cgstreetscombined by segment_id to derive the geometries and street names.

sdpavingdatasd

This is the full work layer, encompassing work done from the beginning of FY14. We’ll talk about the specifics in the data delivery section. This has no geospatial data. As layers get selected, SQL is generated, applied to this layer, and joined against cgstreetscombined.

oci2011datasd and oci2015datasd

These were created using historical data provided in the Pavement Management System and the new OCI numbers we received from latest run of the most recent vendor. These also have no geospatial data. As layers get selected, SQL is generated, applied to this layer, and joined against cgstreetscombined.

Data Discovery

We partnered with the Transportation and Storm Water Department to build the map. However, we quickly ran into some challenges with understanding the data, and the calculations performed for reporting.

The PVM (let’s use that as the acronym for the Pavement Management System) is an application that is built on top of SQL server using queries and stored procedures. Every month, the streets team has to update IMCAT. IMCAT is a system the City uses to coordinate street work across departments. This way we can avoid the same segment being dug up multiple times in a short time span. The streets team pull a report from the PVM generated by a query and a stored procedure). Then, it’s manually cleaned in Excel (we counted 32 steps), joined with a streets shapefile, and uploaded it to IMCAT.

This is where we started. The data that gets uploaded to IMCAT has to be in a particular format with certain columns, so we couldn’t directly trace that dataset to the query running against PVM. And we couldn’t just guess the query and match the result either — PVM has 319 tables.

A few months ago, the City invested in a tool called Alation exactly for this type of use case. We wanted to be able to maintain data catalogs and keep documentation on the data that we have, allowing data users to be better informed. This database is the first one we connected with Alation.

We built a robust query to continuously pull data. To do this, we used Alation to trace the query the PVM application was running. However, we had no visibility into the stored procedure. To bypass this, we combined information from using the query, the streets team and Alation’s query logs.

Query Log

Tables

Table Internal View + Our Notes

Built Query for Paving Data

Built Query for Street Info

Going through this process gave us an enormous amount of knowledge of street paving information, which we have documented in Alation.

However, we weren’t done yet. We still needed to build the ETL for Street Map Data, IMCAT data (because friends don’t let friends clean data over and over manually), and the CGSTREETSCOMBINED file.

Data Delivery

We have two main queries that pull data without doing much cleaning. We wanted to stay away from cleaning in SQL as a best practice.

streets_base gets pulled by FME, combined with the geospatial base file, and outputs as a shapefile to S3. Carto syncs it down from there:

Streets Base ETL

R pulls Pavement Ex in using Alation’s API and the Alation R Package, completes the 32 manual steps the department normally has to do (with some conditional forks for IMCAT), and then outputs the sdpavingdatasd.csv file into S3. We also make sure to standardize the outgoing data according to our Technical Guidelines. At the same time, we output the IMCAT file to be ingested for the conflict mitigation map.

Verifying Calculations

So, how in the world do we know that our constructed SQL on the client side is doing the right thing and making the right calculations? We use R to verify that the calculations are correct by pulling the data straight from Alation, and duplicating the calculations that the SQL does on Streets Map.

What’s next?

This is an alpha project, and we plan to continue working on it. We will work in sprints. As people submit feature or bug requests, we will prioritize them according to what we think we can do during the time allotted. Critical bugs always get fixed first.

Of course, we are more than open to Pull Requests. If there is a feature you really want and we’re just unable to get to it, feel free to discuss it with us in the issue queues, fork the code, and make a Pull Request.

Wikipedia’s Definition of a Dataset: > Most commonly a data set corresponds to the contents of a single database table, or a single statistical data matrix, where every column of the table represents a particular variable, and each row corresponds to a given member of the data set in question.

And data.gov’s definiton is: > A dataset is an organized collection of data. The most basic representation of a dataset is data elements presented in tabular form. Each column represents a particular variable. Each row corresponds to a given value of that column’s variable. A dataset may also present information in a variety of non-tabular formats, such as an extended mark -up language (XML) file, a geospatial data file, or an image file.

Nothing is incorrect about these definitions, but they tend to miscommunicate and under-explain what a dataset is. This causes a lot of confusion and frustration in the government open data community (and probably other communities as well). People end up thinking that there are datasets laying around on a shelf somewhere — that it’s a finite, well-defined thing that a city employee can just grab and publish. Sometimes, it’s true. However, more often than not, a dataset is more like this:

There are two properties that every dataset has that are critical to how useful it is — format and structure.

Format

Format is fairly straightforward. In order for a dataset to be used by as many people as possible, it has to be open and machine-readable. This is just a primer — I’ll go into more detail in a different post.

Open

Open formats are those that can be read by a simple text editor. For example, CSV files — a common format for tabular data can be opened in Notepad. They will be hard to read by a human, but it’ll work. They can also be opened in Excel and plenty of languages have CSV parsers.

However, the sister of the CSV file — the Excel file can only be opened in Excel. That means that if I want to look at your file, I have to give Microsoft some serious $$ for Microsoft Office.

Machine Readable

The other aspect of an open dataset is that it’s machine-readable. This ties in with “open” — since many more languages and libraries can read formats not copyrighted by a vendor. But this also means structured in a predictable manner. For example, a PDF file is not machine readable because it’s digital paper — and breaking down the structure of a PDF is really hard for a program that can only deal with structured data — the format is closed and copyrighted, and also extremely opaque.

Structure

Structure is a lot more difficult to get right than format. For the sake of this post (and to keep you awake), we’ll talk only about tabular data (data that lives in rows and columns). There are other types of data though, such as GIS (mapping) data, XML / JSON (nested data), and several others.

Let’s step back and take a look at a database (or the dough blob). We’ll keep NoSQL databases out of this for simplicity and only focus on the old-school relational databases. What does a database look like?

Contrary to what Hollywood would like us to think, this is not a database:

A relational database isn’t really that exciting — it’s just a collection of tables (sorry, I tried, but relating this to cookies is hard):

These tables contain a variety of columns. Invariably, one or more of these columns is going to have a set of values that relates to a set of values in another table. That’s what makes a database relational and also what gives the technology so much power and flexibility. It’s like dough — you can shape it, mold it, cut it and change its shape as much as you want to fit your cookie idea.

Let’s take a look a more concrete example:

Let’s say you have a company that sells widgets. You keep track of your customers, and you keep track of the widgets you sell. Let’s imagine a very simple database with just three tables:

There is one table to keep information on your customers, another one to keep track of the widgets you have, and a table that keeps track of the orders — where the customers bought the widgets. For the sake of simplicity, there is only one customer per order and one widget per order.

Let’s look at some example data in this type of structure and examine some points below:

• We can see that each table has an ID column. This is called a primary key and is used for the database to differentiate each record in that specific table. A lot of times it corresponds to a row number.

• In the orders table, we can see that each order has its own ID but also references the ID of the widget sold and the customer to whom it was sold.

So with the following example, what are some datasets we can generate? There are many options:

• We can get 3 datasets already just by using the tables themselves. So, there will be a customers dataset, a widgets dataset, and an orders dataset. However, the orders dataset will be pretty useless, since it references IDs only relevant in the context of the other two tables.

• We can get a list of only those customers who have purchased a widget.

• We can get a list of customers who purchased a widget and live in Chicago.

• We can get a list of all the customers regardless of whether they have purchased a widget.

• We can get a list of customers, the widget they each purchased, the model of the widget, and how much the widget costs.

• We can get a list of widgets and which customers purchased them.

• Or a list of widgets and in which city they were purchased.

We get these datasets by using SQL, a database query language, and what you see above is not an exhaustive list: there are many other alternatives. In addition, due to data schema changes, process changes and technology changes, data after a certain date may not be valid or may not mean the same thing. For example, up until our widget company changed their process in 2011, the “city” column in the customer table indicated the city in which the customer was born, not where he or she lived.

This complexity is present in our simple fake database of just three tables, but modern relational databases have upwards of 100, often thousands of tables. There are also plenty of caveats in how they’re named, how relations are structured, and what type of queries make sense. Plus, writing the queries is no easy task.

Some Clarification

I want to take a moment to clarify here — I’m not saying that every dataset is hard to create. Sometimes it’s as easy as grabbing an Excel file from the desktop. But even then, it’s still necessary to make sure that there are no sensitive data in there, such as someone’s phone number.

I also haven’t touched on things like aggregation and making sure that we generate data that fits the Tidy Data Spec so that it’s prime for analysis.

How do you define a dataset?

I hope I was able to show you that it’s not all cut and dry.

The next time you go and ask someone for a dataset, bring them a box of cookies just in case. Fulfilling your request may involve a lot of hard work.

Originally published at quandary.io on April 20, 2016.

Fairly recently, I celebrated my 1 year anniversary of being with the city of San Diego as the Chief Data Officer. Naturally, I’ve been reflecting on my first year in government, and one of the things that strikes me the most is the amount of things that I learned.

Some things I learned on my own — in many cases I invested in supplementary learning in order to succeed. Others I can directly attribute to conversations with my boss, Almis, my teammates in Performance and Analytics, the City’s IT staff, and so many others that if I list them all, this would be a spreadsheet, not a blog post.

To be completely honest — I had my hesitations about taking a local government job. I was worried I wouldn’t advance my technological skill, and potentially lose many of the marketable skills I possess. However, looking back on this year, taking a government job has proven to be one of the smartest and challenging things I’ve ever done. There is no other place I can think of that would push me so far out of my comfort zone, in so many different directions, while providing me the help and support to succeed.

Technical Skills

Tools for the Job. I’m proud to say that I have a very wide expertise in a diverse range of technologies, languages and stacks. I walked into this job with a preconception that my large toolbox can solve a variety of problems quickly. Instead, I quickly realized that in order to use my toolbox effectively, I needed to understand all the intricate non-technological complexities such as strategy, culture, processes, and even words used to describe things. What’s more, I had to expand my toolbox with more open source and not-so-open-source tools. The City is a large enterprise, and not everything can be solved by cloning a repo or a $30 a month subscription. This intangible skill of understanding City operations and figuring out the details of how to apply the correct technology has been one of my biggest learnings this year.

R Language. This past year, I learned the R language — the first data-focused language I’ve used. In addition to learning the language itself, I learned how to communicate research and data analysis through detailed and reproducible reports.

Google Apps Script. I wouldn’t necessarily say that I “learned” it, but I definitely became much more familiar with the capacities of Apps Script for handling large amounts of documents programmatically. This code ran a lot of our initial data inventory.

GIS. Through our wonderful GIS manager, ArcGIS and CartoDB, I became much more familiar with GIS operations and concepts. However, I would place myself very far from expert level.

Cybersecurity. Something we often don’t think about when it comes to cities is the massive need for cybersecurity. Cities are constantly getting hit and need robust and well-managed cybersecurity programs in place. I have become good friends with the CISO and the Cyber team in San Diego and learned quite a bit about their strategies, technologies and methods.

Enterprise IT. We often like to make fun of “antiquated” City technology like SAP and Oracle. However, these are still massive companies running multi-million dollar projects and are not going away soon. I’ve worked quite a bit with IT to understand the ecosystems and infrastructure that manage these large datastores within the City.

So Many Datastores. This brings me to the diversity of datastores within the City. From a technical level, I’ve been exposed to a variety of SAP, Oracle, Microsoft and ESRI products. This year, I’ve had lots of technology land on my plate. Even though this isn’t very sexy, going down the journey of learning these technologies has been a blast.

Networks. Working with IT to understand the City’s networks, I’ve learned a ton about the City’s DNS architecture, networks, configuration management, desktop management, Active Directory and so much more. The City also does a lot of cool things, such as producing its own TV station. We even have a truck that can pull up to a location and provide wi-fi and cellular communications in emergencies.

Strategic Skills

Technological Interventions. Operational improvement opportunities that would otherwise remain hidden often surface when a public-facing data visualization or analysis is on the horizon. It’s a fortunate thing I work in a department that cares to fix those as well. I have also begun to explore how to use what I call “technological interventions” strategically to discover problems in processes of collecting, generating and sharing data across systems.

Communicating with Executives. In the course of my work, I had to learn how to work with elected officials and high level executives. I learned how to properly scale the level of detail while explaining complex technical concepts and challenges. Scaling detail is extremely important because it’s often necessary for me to walk a fine line between confusing the person I’m talking to and providing them with the amount of information necessary for them to make a decision.

Managing up became very important as well. My boss is extremely flexible and understanding, but oftentimes he doesn’t need to (or really want to) be deeply involved in the details of my decision making and thought processes: he empowers me, which is both scary and great. It makes sense too: he really doesn’t need to know all the variables I’m weighing when deciding on a technology or working on a project. By managing up, I’m able to keep his expectations at the proper levels while giving myself a good amount of flexibility to change my mind.

Managing Meetings. Government is notorious for having a meeting-loving culture. I learned how to manage and limit the amount of meetings that I end up involved with, while making sure that I maintain relationships and remain apprised of what’s going on.

Evaluating technology in the context of the City, or rather in the context of enterprise IT requires weighing many factors. I learned a lot about how to evaluate technological solutions in this type of context. How does a piece of software fit into the larger IT roadmap? What do the service level agreements and maintenance options look like? Is it better to get Open Source or proprietary solutions? How will this impact the City’s cybersecurity risk exposure? What is the price point and what are the related procurement rules dictating the process we’ll go through? All these factors weigh into evaluating a piece of software in a large IT environment.

Hacking People, no USB cable required. No, really, I learned how to get 65 non-technical people who have other jobs to answer my questions about what data they have, make it extremely easy for them to do so, and not have them hate you afterwards. Oh, and only have one large meeting through that whole process.

Shipping. I learned quite a bit about how to balance the inherent complexity of being a CDO of a massive organization with designing the correct architecture, building a sustainable infrastructure, and well, shipping.

More Shipping. Elaborating on the point above, I tend to lean more on the perfectionist side of being a programmer. I’ll think deeply about choosing the right framework, picking the right stack and overall making sure I’m doing things “right”. In the past, this caused unnecessary slowdowns in some of my projects. However, in this past year, when I had a much smaller part of my time to be a developer, I noticed myself being able to adjust my “perfectionist” strategies towards a strategy where I just write some code, and refactor if I’m going to keep it. It seems like a minor shift, but being able to accept imperfection in my own code has allowed me to prototype a lot more, a lot faster.

Technology Perception Gaps. I’ve learned to see and more clearly understand the gap between the public’s perception of Open Data and the technology and effort needed to deliver data that isn’t just open, but is accurate and complete. While this gap exists in the Open Data space, I would argue that this gap exists in many perceptions that the public has about City IT.

Learning from Private Sector. I often seek out sources of inspiration for my work by attending private sector events, training sessions, and simply communicating with technology communities that I’m used to working with. I also love to interact with other governments to see how they’re doing things. However, not everything that works in the private sector works in the government space. I learned quite a bit about how to learn (and not learn) from the private sector and relate it to our work at the City.

Open Data Activation Points. I’ve begun to see “Activation Points” for open data — where opening data ties in smooth as butter with a current City initiative. (Sustainability, Performance Management, Public Records Act Requests (PRA)).

Building and running a data team is a leadership, strategic and tactical challenge. I have learned, and am still learning a lot about how to build one, ramp it up, and align the correct pieces for delivery.

Departments / City Operations

Lifeguards. I learned a ton about what lifeguards do, how they operate, and some of the technological challenges of the job.

Police. I explored how the City’s Police Department works with bar owners in the Gaslamp to create a safer area, handle barbreak (when all the bars let out) and prevent crime.

Street Work. I know much more about how the City’s street work gets done, and the different types of repair jobs. I was also surprised to find out that the City does a proactive job of making sure that money isn’t wasted by having the same street repaved multiple times in a short period. Because of various responsibilities different departments have when it comes to digging up a street, they do a lot of work to coordinate the timing of the construction.

PRAs. Many people outside of government or journalism don’t even know about PRAs. Working with the City’s PRA coordinator, I’ve learned a ton about the legalities of the process and how they work within the City.

Emergency Response. Do you know what to do if an earthquake hits? Your City officials do — they often conduct training and simulations for different disaster scenarios. I got to explore this as well. They have a “situation room” in one of the buildings, stocked with food and computers. Learning about the City’s ways of managing disaster has been nothing short of amazing.

Crap. I learned all about what happens to the crap we flush. It’s actually way more interesting than it sounds. This is a story that encompasses poop, energy generation, earplugs, and seals. Where did I learn about this? The Point Loma Wastewater Treatment Plant — an automated plant that serves 2.2 million residents run by only 50 people.

Communications and Community

Talking Louder. I have been a quiet and shy speaker for most of my life. However, when I started this job, I quickly learned (with Almis’ advice) that I needed to overcome that. I took several months of improv classes (which were a blast) and while I’m still working up to having a booming voice, I can definitely say that I haven’t been asked to speak louder in a long time.

Press. I learned a lot about how to talk and work with the press. It was really fascinating to learn the role of press in government and how the City handles communication with journalists.

Public Speaking. I still need to tally up how much public speaking I’ve done this year, but it was definitely more than I’ve ever done in my life. There’s plenty of things I learned in this arena this year, such as structuring effective presentations, storytelling, and other skills.

Ignite Talk. I gave an Ignite Talk about the data flowing through a City block. Obviously, I learned how to give an Ignite talk, which in itself is a very difficult, but ultimately rewarding experience. But through that process I discovered how much I knew about the data flowing through cities — and how much I have yet to learn.

Reporting to Council. I also got to prepare a report to the City council and present it — a learning experience in and of itself. As a side note, as part of getting the Open Data Implementation Update ready for council, I ended up making a pull request to a major open-source software project.

Community Building. I always felt that in order for Open Data to be successful in San Diego, we needed to have a strong Civic Tech community. Before, this was far from the case. Now, we’re much closer — I’ve learned a ton about how to put the right people in the right place to get the community moving. I’m very excited about the work Open San Diego has accomplished in the past year.

Connecting Geeks to Government. In addition to building the community, I’ve always strongly felt there was a missing connection between CfA Brigade members, residents, and City employees. Bringing various people in the City-people I greatly respect-to talk about what they do with the brigade has been an invaluable learning experience both for me, the brigade, and the employees. I think we often forget how freaking cool City government can be.

Surprises

Passion. I was surprised about how I feel after a conversation with a City employee (not everyone of course, but so many of them). For example, I talked to someone about parking meters and was inspired and excited after that conversation. It’s amazing to see people that have just as much passion for their craft as many programmers do for theirs.

Data geeks. I’m not talking “analysts”. I’m talking about the people that geek out over Excel macros and SQL queries. Or GIS. There aren’t many of them in the City, but they’re there. It takes a while to find them too. But once you do — wow — they can teach you so much and are such a pleasure to work with.

Data Literacy. How people think about what data is, how they define data, how they understand KPIs and metrics has been a challenge that I have enjoyed learning about and solving.

Open Doors. Many people already recognize the potential for decreased workload and better communications that opening data carries. I was extremely surprised how many people approached me, excited for their data to be opened up.

State of The City. Did you know that just like the President does an annual State of the Union, many mayors do a State of the City Address? I didn’t. It was amazing to see it get put together, how many people work on it, and some of the things I learned by attending. Also it was pretty sweet to get a shout from Mayor Faulconer just a month into the job.

Dependencies. I noticed a distinct difference between how I’m used to working to how I have to work in the City. In my previous jobs as a software developer and architect, I could generally forge my own path and minimize a lot of critical reliance on other people. In the City I feel highly leveraged — oftentimes the support of others is absolutely critical. I haven’t thought too far down this path yet, but will be exploring it.

Internationalizing. I’ve had the opportunity to present to Macedonian and Philippine delegations. In the Mayor’s conference room. About Open Data. Nuff said.

CNAMEs. When we were launching OpenGov — our budget tool, I thought it would be great for us to map it to http://budget.sandiego.gov. The saga that happened while I was trying to do something fairly simple — just map a CNAME to a domain — was a journey I will never forget.

Random

Vendors. Working with vendors has been good, bad and ugly. There is so much to write about here: from evaluating proposals and negotiating; to the “classifications” of vendors I’ve begun to unofficially define based on what they sell and how they sell it; to how they can have better strategy and products. We have developed numerous ways to streamline the process of vendor pitches by having a standard method and questions that they have to use to get in touch with us.

City Awards and Competitions, run by various think tanks and vendors are also something I’ve been reflecting on lately. While having some value, a lot of times they appear to be run more for the benefit of a vendor looking to score more government contracting opportunities.

Time Management. I’ve had to implement rigid time management methodologies because of the amount of noise often coming on my plate. While this may seem like an underwhelming learned lesson, I would say this is what has allowed me to become the most effective.

Project Management. When you have an organization of 11,000 people, IT projects quickly start to look a little different. Sometimes there’s a legitimate need for a strong project management methodology. Sometimes there isn’t. I’ve learned quite a bit about how the City runs large IT projects–and small ones.

Hiring. As part of hiring my first employee, I got to be on the hiring side, all the way through the recruitment process. I ended up taking notes for myself, for, you know, the next time I have to look for a job. It was super valuable.

Supervising. Because I was getting a new employee, I had to go through the City’s “Supervisor’s Academy”. This was a seven day course, outlining everything from City support for employees experiencing various issues, to ethics, to leadership. I found it extremely useful and learned a ton about, well, being a good supervisor.

Ethics. I learned a bit of how ethics and campaign finance work in the City context, and had some fairly interesting conversations with the Ethics commission, including one about the limits on what expenses can and cannot be covered by a conference organizer who invites me to present at a conference.

In summarizing my experience at the City, the main thing that surprised me this year was how much I was pushed out of my boundaries, and how many things I’ve done that I never thought I could or would do. Everything from learning a new language, doing an Ignite talk, to learning how to be a better communicator and leader, I can attribute to being a City employee in one way or another. Now that year 1 is behind me, I’m stoked for year 2.

Originally published at mrmaksimize.com.