Templates!

Ah, these kids today call a Website design a template.... Here's one. Here's a basic approach.

DotCloud

Interesting article. DotCloud is a multilanguage cloud hosting service. Good Sproggler target.

LRNJ

Learn to read Japanese while playing an adventure game.

Wesabe's code

So Wesabe was a service to allow some kind of personal finance management - it failed. But they put together a pretty comprehensive set of code to slurp bank information out into a local database for you, and - lo! - they open-sourced it on Github. Here's an article about it.

Food for thought, if I can figure out how to make use of it.

Two NLP libraries/APIs for the Web

Pattern is an NLP Web mining module for Python. It makes my mouth water. And Repustate apparently is SaaS that does something with sentiment analysis. So my mouth just keeps watering.

jMockups

This is cool: it sucks in a Web page and makes a code mockup you can work on and share. Pretty damn close to what I want to do with that end of Sproggler.

PHP examples

So I'm looking for some really basic PHP examples to use for initial test cases. Here's a link dump with a few - it's surprising how translating some of this stuff into Declarative for publishing motivates the design work. Moral of the story: more examples! Lots of little apps!

• Tizag form example
• PHP-learn-it
• T. David's PHP script diary - older than dirt but still not a horrible way to get used to the language
• Well House Consultants has an example of uploading a file for storage as a blob in a database. Kind of a rudimentary document management feature. Neat, actually.

But those are really only for getting off the ground with PHP. Nobody writes stuff like that any more, it appears. They're really just good for a couple of proofs of concept. I've taken eight pages of paper notes on thoughts about just one signup form using jQuery; I'm getting pretty close to being able to start coding - so much for getting Depatenting done by the end of the month to be able to use the Google Adwords coupon I got.

More on this topic later.

(Later: also a list of open source PHP script modules. Either integrating these or analyzing them to distill commonalities for some basic modular features would be the way to go.)

Scaling

Scaling. It's what's for dinner.

There seem to have been a lot of thought-provoking posts on scalability this week. Here is another area where a macro system makes sense. It's nice to have a simple prototype that doesn't need a whole lot of infrastructure, when you're hacking out an idea. But if that idea gets featured somewhere and it's written as a low-traffic app, you will suffer.

A good macro system will be able to bridge that gap, keeping the clean lines of the code understandable while allowing memcache to be shoehorned in all over the place.

This post is a link dump right now, but I want to come back to this topic and come up with an overall approach that works to describe scalable systems.

First, a great talk by Steve Huffman on lessons learned at Reddit. There are a lot of ideas packed into this talk; I've listened to the talk but haven't unpacked the ideas yet.

Second, a much simpler post from Dan Shipper about WhereMyFriends.be, featured on Mashable and CNN and thus buckling under the strain.

Programming tricks for memcached. Lots of thought needed there.

Photon, the high-performance PHP framework.

Prototyping vs. Photoshop design

Couple of articles this week about doing your design in code instead of Photoshop. I'm thinking it would be cool to have a Photoshop-like drag-and-drop prototyping tool, of course, as part of Sproggler.

First was a post about the Quora design process, no less. This was followed up by an article by Chris Teso elaborating on why he agrees. Using the neologism "devigners". ...

Anyway, it's a good idea. And while I'm on the topic of design, a slight link dump: improving a specific design, a step-by-step case study.

The difficulty of best-practices code patterns

OK, so now that I'm thinking more carefully about macros (in the context of publishing Javascript for Web apps), here's why they're important: because it's too damn hard to understand code when it's written to work right in all the fiddly-bit situations.

Case in point, this article about asynchronous code patterns in node.js. Go ahead, read it.

Now here's the thing. If you read that code, the first version, the synchronous version, is comprehensible. The final version, an asynchronous version that actually works and handles exceptions rationally - well, you can understand it given the presentation, but I defy you to run across that in the wild and tell me what it does, clearly, without the need for fifteen minutes of error-prone thought.

Macros help us get around that. If we think of that final pattern as the compiled version of whatever concept we're trying to express, then it's a lot more palatable. And that's why a universal macro language like Declarative is going to be really useful for a lot of different things.

SDL

Another interesting GUI/graphics framework: SDL. Specifically, the Perl SDL game contest. This sounds like a fun challenge! And SDL::Declarative practically writes itself.

Algorithms

There are conceptually (for me, anyway) two different kinds of programming problem. First is the "set it up and let it run" kind that Declarative mostly addresses - the construction of data structures to model and simulate a particular system.

The second is solution algorithms - code that takes a particular question or problem in a problem space and derives a solution. This stuff often verges on artificial intelligence, and Declarative mostly doesn't address it. Nor should it, arguably - Declarative is intended to declare things and remove the hassle of setting all that framework stuff up. You can implement your algorithms in whatever language you like within that framework.

Anyway, this came up because of an interesting post on dynamic programming.

"Base programs"

Grist for the mill of "types of program" here on HNN, ever the most interesting forum out there.

The list:

basic form
form with multi-valued lines
form with data set (multi-valued parent/children)
form with grid
tabbed form
form with skins
batch (loop thru something)
batch update selected records in a table
batch dump table records --> .txt or .xml file
batch .txt or .xml file --> update table records
batch file --> create & populate database table
cut an email
traverse internal tree function
traverse file index(es)
build html from parameters
build javascript from parameters
build .pdf from parameters
build hp esc sequences from parameters
benchmark a process
how does that syntax work?
which way is faster?
batch string parser
batch source code search
batch source code changer
batch parse source code, identify routine for reuse

A follow-up comment mentions that these should be in a library - but I beg to differ. This is a higher level of organization than the library.  This is where we talk about the semantic purpose of the program.  It fits well with the Unix philosophy of many small utilities, actually.

Food for thought!

Node.js as part of the "standard stack"

Scalability in Node.js, which is pretty damn cool.

PhiloGL

A WebGL framework (OpenGL in the browser!)

Dysfunctional: LaTeX in Javascript

This is mindblowing.

Web app design

The realm of Web app design is really pretty big. I've worked with it since the 90's, of course, but ... a lot has happened rather recently, and I'm still only scratching the surface. I think it's safe to say that it will continue to change.

The underlying structure of any Web application is, of course, a set of files on a disk somewhere. So at the lowest possible level, when specifying a Web application we're actually specifying a set of files. A structured set of files is a product. I'm going to call it that for now, because I want to drive home the idea that the notion of a product is a more general thing - it's a thing that properly belongs in C::Decl, not in a library domain. It's part and parcel of files and directories, because it is a set of files in directories.

So we're going to have a product template structure, and one possible expression of that product template structure is going to be ... well - hold back a little, because obviously there will be a variety of different types of Web app as well (e.g. a LAMP PHP site, a Google App Engine Python site, and so on). The commonality of these types of Web app is that they're made up of pages and supporting files - they all consist of a set of server code in one language or another which generates HTML, and they all include CSS and Javascript.

Earlier today I posted on the "HTML5 boilerplate". This is at least part of what goes into a Web app template. I'd go so far as to say that the basic page template is also part of that macro. And make no mistake; the boilerplate is a macro that we will never see in our specific Web app definition.

Some Javascript and some CSS will also be part of the boilerplate.

The components of a Web app thus become: a specific database and table definitions within that database (aside: some table definitions will probably end up in boilerplate and component macros as well), a boilerplate and sets of imported components, some basic publishing actions to be carried out for different stages of the game (which could also go into boilerplate, actually), and the list of pages/application areas that will show up in the final application.

The only way to work this out is to start building Websites. My initial goal is the Depatentizer (8 days left, haha), but some smaller apps might also be a good idea. And then I want to rebuild Despammed, and maybe even Techspex.

I've been prowling around looking for insight, and so the rest of this post is a link dump about PHP site building (mostly).

Oh, wait: a mother lode of so-called "transaction patterns" used in a toolkit for building database administration sites. It is incredibly detailed. These transaction patterns, by the way, constitute a language for user-level interaction with Websites that is not restricted to the Web domain - they'll work just fine in the Wx context as well. So they should definitely be analyzed, but they can't just be relegated to WWW::Publisher. I'm not really sure where they should go. Probably a new domain. Database::Declarative? I just don't know!

Anyway, that link is brilliant. Brings up the MVC architectural pattern, too - again, this is a solid concept that should be highlighted somewhere, but not in WWW::Publisher. The link there is to the same site, and it includes more than MVC; it's an overview of his entire methodology, some of which I don't care for, but still - wow! Great find. (Building up to a PHP sample app, which is where I found it.)

Now for the link dump.

• DBI for PHP is PDO.
• CPAN for PHP is Pear: Pear coding standards.
• Writing web apps in UML... Mentions WebML (the Web modeling language) - neat!
• Units of Work (UoW pattern) in PHP.
• AJAX overview article. Haven't read all of it.
• Oh, oh, oh - this is also brilliant: a blueprint for PHP Web apps with a mindmap!
• A follow-up article from that series (which mostly never got written, sadly): the bootstrap. This would go into the boilerplate.
• Flourish library - most interesting to me for its abstraction layer over a variety of databases, allowing you to write a Standard Query Language that will be munged into what the specific database needs. That deserves some brain picking.
• Smarty is a PHP template library. The link is a sample app.
• Now, for something completely different: MVC setup in Node.js server-side Javascript. Now technically a Web app definition should be relatively identical whether we're writing for PHP or Javascript on the server, so .... this also needs thinking about.
• Canvas whiteboard with chat using a web socket. That is a collaborative tool that would be useful on sproggler. And any tutoring site.
• A thoughtful article on prototyping and customer immediacy in the early days of a site. Some components for quick feedback should go into the boilerplate.

So there you go. Lots and lots of material to digest. Remember: the only way this will work is to write lots and lots of little Websites. I'm thinking of following that blog with Webapps on it, and doing as much or as little of each as I want to in one sitting, say. A lot of the components of these Webapps should break down into components rather easily, and then I should be able to snap together a site in minutes!

Scheme in Perl

Some useful notes on more Scheme-y stuff in Perl. Pretty cool stuff.

Wampserver

Useful all-in-one package for MySQL and PHP under Windows.

More on Web design

10 examples of page layouts - useful for prototyping. And designshack gives us 10 Google font stacks. Meanwhile, 10 useful Web application interface techniques.

HTML5 boilerplate

Or, "how to get a project off the ground". Paul Irish came out with a definitive one. Harry Roberts(!) came out this week with a minimal one. The point: this is a "site template".

Nice drop shadows

With CSS. I really want to look at CSS carefully one of these days.

The Semantic Web

Looks like this is starting to gain some momentum. I'm too tired to do anything with it just now beyond pointing and screaming, maybe hitting it with the thighbone of a dead animal - but later, boy howdy, later I'm going to understand the hell out of it.

Field - sort of a Pythonic Processing

Or something. It lets you make graphical representations of your code. Jesus Mary and all the saints, that is just stopping me in my tracks here... Literate programming done really right.

This is what I want for my birthday, Lord. Only in declarative wxPerl.

FluidInfo

Hmm. A writeable API for data from BoingBoing... This seems to be some kind of scraping/semantic Web database thing that permits the addition of hierarchically organized data by multiple users. Kind of neat!

Morphic.js

A Javascript port of Morphic, the GUI toolkit of Squeak.

Nemerle

A .NET-based programming language that has pretty nice macro facilities. Worth studying! I wouldn't have heard about it had Wikipedia not decided to delete its page due to lack of notability.

Keynotopia

Keynotopia is a quick startup that provides Keynote and PowerPoint templates for fast prototyping of apps - publish to a PDF and you get a prototype you can actually run on a phone. Which is a cool idea! So I want to do that with a more advanced prototype based on Decl.

Using public data to fight a war

Military contract to data analysis company, film at 11. Good make-you-think article.

Visual Website Analyzer

Target application.

Various diagramming things

Oooh, started with inubit, which is closed-source but has features that make my brain melt. This. I need to just scan down the menu on the left.

Then there's Pic2Plot-gui, at Gitorious, a Qt-based UI for generating PIC. For which latter, see.

Also: GraphViz.

Various open-source Project things

• GanttProject is a desktop tool (Java/Mercurial/ANT build).
• DeskAway is a Webapp (not open)
• A whole list of alternatives
• MPXJ, a Java library for reading Microsoft Project files and writing some of them

MySQL explicit query strategy

I'm not sure how best to go about codifying this sort of knowledge, but ... it's an interesting short read.

Congressional sentiment

NLP judging Congressional sentiment, from Cornell. I just need a quiet day to read it, I suppose.

Hmm: Java webapp framework JWt

JWt uses the class structure of a Webapp in Java, and "the library generates the necessary HTML/XHTML, Javascript, CGI, SVG/VML/Canvas and Ajax code".

Sexy!

Javascript goodies

• A look at grid controls/spreadsheets in Javascript.
• Rails AJAX frameworks - best practices at the moment.
• Joint.js - charts and diagrams in Javascript.
• HTML5 canvas graphing solutions - Webdesigneraid might be something to follow.
• jsfx sound effects library in JS! Very fun!
• LimeJS game library.
• mxGraph - not open-source, but very, very feature-full. Good for inspiration.

Building block database

When looking at an overall system ecology, we see a lot of applications in it that cover certain areas or regions of functionality. To a certain extent, those applications are interchangeable. What I'd like to do is to index some Web apps and open-source apps (see osalt.com for an example of this at a coarser grain than I mean) and reverse-engineer their specifications in some way that facilitates comparison.

Then, when putting together a specification list for a given business process (say), you could choose which elements to plug together, and identify places where interoperability projects might pay off.

I'm still not 100% clear where I'm going with this, but it deserves further thought.

Target applications: design-to-specification, also Web app reverse engineering

This seems to be an interesting little niche: taking a design and spitting out a framework for coding. Obviously it's not automatizable, but what I want to be able to do is to specify such an app in a simple way (WWW::Publisher, of course).

Also: I'd really like to be able scrape a site of this nature into a specification for analysis and modification.

31 CSS code snippets, OOCSS

More CSS information for my later reading delectation.

Also: an object-oriented approach to CSS construction. Very interesting!

TheJIT

Interactive data visualization in Javascript.

Isotope

Animation framework in jQuery... Argh! Too much cool stuff lately!

DataMarket

Oooh, data.

Ninja - a new build system from Google

Now this is cool - minimalism always cuts closer to the semantics of a domain.

Ooh - also gyp (Generate Your Projects), a "meta-build" tool.

jQuery runloop for animation

I just have the feeling that Webdev has undergone a phase transition since I looked at it last...

Animation in JS.

Awesome Web fonts

This ... is pretty neat. I need to understand the CSS end of it.

AppVita

AppVita is a blog devoted to Webapps that do useful things. This is an incredible source of grist for the "Do that!" mill in the WWW::Publisher domain.

Automating Windows

So today I'm scripting the conversion of about 140 XLS and PPT files scattered in a directory tree on FTP into TTX in the same FTP directory tree. There's no way to do this except by calling an external Windows function and doing your best to automate it through the GUI.

This is a brittle process, and therein lies the difficulty.

Several of the XLS files are locked in Excel. TagEditor can still convert them, but Excel pops up a little dialog asking for the password - and since the TagEditor window never appears, the script runs afoul of my assumptions.

The way to get around this would be some sort of wrapping layer that explicitly states my assumptions and ensures they're met before proceeding. If something unexpected happens, ideally this facility would note it and diagnose it in a wait-a-minute record, and put the action back on the queue for intervention.

Which brings me to the other assumption I'm making, which is that a script is a sequential process; before one file is done, I don't go to the next (except of course that the script is sometimes not waiting for some reason, thus opening hundreds of TagEditor windows without checking whether it's appropriate or not, sigh.) Instead, there should be a parallel queue (this is a workflow concept); if one of the actions runs aground, others should proceed.

Both of these are higher-level approaches to the scripting task.

(Update: additional assumptions I was making: that the window closed immediately after being told to close, that the converted file was being written immediately, and that a window just opened is the topmost window after opening. All of these should be checked and accounted for in a mature script driver for Windows.)

GNU utilities re-implemented in Perl

Freaky. What a fantastic resource!

Trees in pure CSS

Neat trick.

PHP

I'm targeting PHP with WWW::Publish, because it's ubiquitous.

Coolly, you can run it from Perl: with module PHP. This is going to rock.

Scraping

I have a spider app in mind. I was DuckDuckGoing on Toonbots, for whatever reason, and ran across a couple of links to "Toonbots forum, being able to avoid spammers and trolls and whatnot." I vaguely recall that conversation.

Well, it turns out that that sentence was incorporated into a whole series of spammy landing pages inserted all over the web, pointing back to e-loan.expert.com via Javascript redirect. This has been a couple of weeks ago, so many of these are getting rolled back up and fixed, but ... it would be absolutely fascinating to make it a statistical project.

I know, I know, I'm a sucker for Web spidering, too. Sigh.

This is how it would work:

• Seed it with one or more of the target sentences.
• Google a sentence.
• Try to find its actual origin.
• Store all the other URLs and text; break the text up into sentences.
• Spread from there.
• Index all the Javascript, in case techniques varied.
• Try to categorize the type of host (different breaking techniques were probably in use).
• Track down everybody making this possible, and fix each and every one of them. Oy.

Wouldn't that be cool?

Seed sentences:

• Toonbots forum, being able to avoid spammers and trolls and whatnot.
• The world is awash in fast money, he said, and it is changing the structure of capital markets.
• When you are pre-approved by spruce mortgage, you will have access to hundreds of loan programs.
• Many sellers would rather have a monthly check than a lump sum settlement when they sell.

Example page: http://cas.ncat.edu/Departments/dance/js/dojo/pbs/ohioautotax.html - hasn't been cleaned up yet! Note that it's inside a Javascript directory for something. This is the kind of thing it would be cool to track down.

Update 11/19/11: That particular link has been cleaned up, but the network as a whole is still there and still forwarding to the same ... actually, just a very similar site. Of course, there may have been multiple sites all along. So this project is still waiting to be done.

Wrestling with SQL for real

I'm down to the wire on table definitions and trying to come up with a good way to combine quick-and-dirty things with more careful solutions. I think the current implementation of the table tag is going to do the trick.

Here's a bit of a link dump.

• A useful comparison of popular field types in several different databases.
• Um...

I guess I only had one link up about tables. Weird; I could have sworn there were two or three.

Mail handling

I just keep coming to this, don't I?

Mail::Box is a hueueuge mess. It doesn't load from CPAN on Windows, and reviews indicate that while it does a whole lot of things right, it makes a lot of very weird and undocumented assumptions that I, for one, just don't care for. So I'd like to do ... something ... for local email handling that starts more or less from scratch. Sigh. Then wrap it in Decl and a UI, and we're off to the races with something nobody cares about.

(I just really want to be able to write Perl filters for incoming mail, dammit.)

So ... Mail::MboxParser looks like a good start, maybe. Unfortunately, it relies on Mail::Mbox::MessageParser, which also assumes a Unix environment that may or may not pertain (CPAN testing fares poorly on Windows platforms - including mine). The reason here seems to be that it wants to incorporate compressed mailboxes using an external gzip. Why is that in the core module? Ugh.

I dunno. MIME::Tools is certainly definitive for handling individual messages, so perhaps I don't even want to worry about anything but that. Even MIME::Tools is horrible enough.

Email is a mess.

Yices

SMT engine. So help me, I'm just a sucker for this type of thing. Microsoft uses it to validate drivers (example). Anyway, Yices is an example. Background. And the trigger - a Haskell DSL for expression of SMT problems. This is the kind of thing I want to be doing here.