Ting wants instant public records


When Assessor and mayoral candidate Phil Ting came by for his endorsement interview, we talked about open government, and I mentioned an idea that sunshine advocate Kimo Crossman first proposed back in 2008: Why not make all city documents (with a few limited exceptions) public the moment they're created?

Why not send a copy of every memo, every email, every contract, every check, everything anyone at City Hall produces, into a public server, where the rest of us can see what our elected officials and civil servants are doing? No more hassles with sunshine requests -- the docs would already be there, in a searchable database.

Well, apparently Ting liked the idea -- and it's now part of his mayoral platform. In a release posted Sept. 13, Ting argues that "everything should be public."

And I mean just about everything. I think that every email, every memo, every check, every contract, every phone message, every tweet, every cell phone call and every other single government document that is not part of an employee personnel decision, about an immediate public safety issue, protected by state law or part of a pending lawsuit should be made public at the time it is created. The reality is that technology has outstripped our city’s Sunshine laws. And it would be far less expensive – and far more productive – simply to have all digital public records (which is now nearly all public records) simply posted to a City Sunshine Site at the time they are created. This site should quickly include, and certainly be the basis of, an Application Programming Interface (API) that gives San Franciscans the tools and the data they need to help hold government accountable.

He explains:

So as mayor, if I send an email to my chief of staff on an issue – that should be made public when I send it. When I have a meeting at City Hall or anywhere else, that would be part of an online calendar, which should be made public. A direct message – a tweet from the mayoral account – just about anything that is created, said or discussed should be made public in real time.

Every document created by city government (with the noted exceptions) should be made available to the public at the time it is created. That should include every check written – and every dollar spent or promised. And every contract. And every subcontract. Everything.

There is simply no supportable reason for any work product created by a public employee to be hidden from the public – or perhaps even worse, to be put behind the barrier of a “sunshine” process that is now so complicated, time consuming and expensive that it promises public accountability without always being able to deliver it.

It makes perfect sense -- the technology exists, and is relatively inexpensive (particularly compared to the time it takes city agencies to respond to public records requests). It would be easy to allow people creating confidential documents (legal strategy memos in the City Attorney's Office, say, or personnel records) to add a tag to the file that would keep it out of the public database -- and, of course, it would be easy for an agency (or the Sunshine Task Force) to search those tagged files later to see what should and shouldn't have been kept secret.

I don't think anyone else has ever done this; San Francisco could be the first city in the country to make sunshine a part of everyday life at city Hall. I hope this becomes part of the mayoral debates.


The problem with that idea is how do you catalog it all?

I'm all for it, but if you go to the controller or budget page its a mess. There is no way to find past budgets and predictions, everything is stored in pdf's and has no standard file system.

for example


doesn't mean you can find that same document for 2000 in the same directory under the same naming structure. I've tried.

The city spends way to much time and energy on producing wind that it would be impossible to create a useful organized system, it will also likely be stored in some PDF or other unsearchable by an engine form and the naming structure may have nothing to do with a continuous undertaking.

Then of course we will need a whole new ever increasing department of flunkies to administer it.

Posted by Matlock on Sep. 13, 2011 @ 4:33 pm

Besides whining about how hopeless the situation is.

Posted by Guest on Sep. 13, 2011 @ 7:20 pm

Have adults run the city for one.

It's not my idea, but I deal with the city, and have tried to look up the past data the city has produced on the cities own web pages. So I know how poorly the city runs and how poorly the city has organized and saves this data.

To save a collate all of this info would be an ordeal, to get the assigned flunkies in each department to follow a standard and coherent formula would be futile.

I'm not whining, I am just stating that in 2011 it would be great if the cities employees could put this all out to the public and in a searchable data base. It would be a costly and futile exercise as after a week or two every memo and report would be buried in a unsearchable .pdf with all the other unimportant make work garbage that the city produces.

Posted by meatlock on Sep. 13, 2011 @ 8:33 pm

I agree whenever reasonably possible a search engine readable version of a document should be created even if you also want to create an image of it as well.

Categories for documents would be useful but let's get the stuff online first - the issue here is to come up with a way to redact portions that must be - probably using a process marking the limited text in the way someone marks BOLD test today. The document with that limited portion would be made public instantly.

There are many solutions to making corporate text based information (not categorized) available as useful knowledge today - The city could issue an RFP for these types of system which can help with the some automatic categorization.

Also the City has a Public Library - They could use their knowledge to provide a reasonable overview of categories for city employees to use for new documents.

Let's not let a perfect categorization standard limit access to so much data we already paid to create.

Posted by Kimo Crossman on Sep. 13, 2011 @ 6:27 pm

I, of course, love this idea and the much shorter Sunshine Ordinance Task Force meetings we would have.

It was a surprise that Mr. Ting could come up with this but it all makes sense that Mr. Crossman first made the proposal.

So, Mayoral Hopeful Ting, why wait to find out if you win the upcoming election? How about getting some discussion going now on this at City Hall in your position as Assessor? Maybe the Assessor's office web site could be the first to set this up as an example. After all, all talk and no action makes Phil a dull candidate!

Posted by Hope Johnson on Sep. 13, 2011 @ 9:22 pm

Making all but exempt records instantly accessible is an excellent idea. Hats off to Kimo and to the Guardian for their persistent sunshine advocacy. At times during the past several years, suggestions from Kimo and other open-government activists have been invaluable to the Sunshine Ordinance Task Force.

Some issues:

1. What would prevent a public official or employee from keeping a given record off the server? And if it happened, how would the public be able to learn about it?

2. Lots of public officials and employees would delay feeding records into the server, offering the excuse of consulting with counsel to determine disclosability, even when they knew that the records were fully disclosable. How could we prevent that?

Posted by Richard Knee on Sep. 15, 2011 @ 9:56 am