Book Review: Perl One Liners

This is a review of a new book by Peteris Krumins, Perl One Liners, 978-1-59327-520-4
oreilly.com/bloggers is where you can get an ebook and review books yourself.
Hurry down to Appendix C, where perl1line.txt is, and shows every one-liner in the
book. Larry probably only needs Appendix C. The rest of us can enjoy the book.
Ever heard of strawberry perl? Or the nuances of PowerShell? There is stuff that
you don’t know in here.

This book can be read very quickly by experienced perl programmers who know about $_ and
hashes and lists and single quotes and backslashes. Less experienced users can
probably use it to get work done fast. Non-programmers and aspiring not-yet-programmers
can probably learn very efficiently using this book. Many of these one-liners are
outright programs that could run to dozens or hundreds of lines of code. The
density can reveal many problems being addressed on one line, including: open file, read line,
process line, do some other stuff, save file. All on one line! Seriously, folks,
CompSci 101 courses might be more fun with a book like this (instead of the standard
C++ tome).

I love Perl. I have long ago moved on to other languages that all the kids prefer.
I don’t buy new perl books because I don’t use perl as often and I’m already proficient
enough to follow the standard procedure for solving a problem with perl:

1. Write down problem.
2. Discover which module to use to solve problem.
3. Read API basics to learn how to use the module.
4. Install module via CPAN.
5. Write program.
6. Go home.

Nowadays, I find myself using python in a similar way. I notice that Perl One Liners
does not avoid using the modules (with -M).

One problem that has been annoying me is migrating data into SQLite. There is not a
reliable program to change a mysql dump into an sqlite database. For some reason,
this problem always seems to require manual intervention. I find myself with 200 character
command lines involving sed and tr. Then I use that output to run through a python
script. Eventually I end up with the SQLite database that I wanted several hours ago.
That isn’t so bad, but when I try to recall how I went from the source data to the
SQLite database, its too tedious to remember or explain. When I saw the new book by
Peteris Krumins, Perl One Liners, 978-1-59327-520-4, I thought to myself that perl
would probably work better than sed sed sed tr sed sed tr sed sort uniq python sed.

I never use Perl on the command line. I remember Tom Christiansen (or Larry?) once
said that you could use Perl as your shell, if you’re crazy. I hope I’m not crazy and
I don’t use perl as my shell. But I have been using it on the command line alot
this week with good results. Using perl one-liners to stuff HTML or mysqldump sql
into an SQLite database hasn’t made things any better. But I probably just don’t yet
have the right line. Thanks Peter! Nice book.

 

Posted in Uncategorized | Leave a comment

Book Review: Android Cookbook

This is a review of Oreilly’s Android Cookbook.  A few notes of disclosure:

1. I only read (and therefore review) books that I’m almost certain that I’m going to love.  
2. I have every Ian Darwin book and think the guy is a technology God.  

Yes, I like the book.  

But the problems get started right away, with Recipe 1.2 “Learning the Java Language”.  This recipe should not be in this book because if you are trying to do Android development and bring flimsy Java to the table, you are not going to have any fun at all.  Java is hard to learn and takes years.  CompSci departments think Java is 3 or 6 credits.  Their graduates demonstrate that this is not so.  The iPhone programming books do this too.  They tell you to brush up on Objective-C and Cocoa.  But they fail to say that “brushing up” is going to take years for someone with no C to start with.  Except for “Learning Java”, I think all of the books recommended are good ones, especially Head First Java, 2nd Edition.  Thinking in Java, 4th edition is also a great book.  I would add “The Java Programming Language” even though it gets very difficult during the second half of the book.  There are a few other bits of context that would-be Android developers should think about:

0. Repeat, Java is hard…be sure to bring a strong Java capability to Android development.  
1. Java and Oracle and open source do not add up to a clear future.
2. Android adopts the new layout-driven programming paradigm.  Its everywhere–HTML5, Rails, iOS development.  Its the kind of programming where tons of things are wired into proprietary XML formats.  Yes, you will need to squint at XML formatting.  XML-wired layouts and XML-wired everything is good-we don’t have to write tricky display code to deal with rendering 8,000 different display possibilities.  But its not fun to figure out whether a <LinearLayout> goes inside of a <resources>.  No fun! Hold your nose and don’t make the display with programming–make it with XML.  You’ll save time and boost quality.

I loved the Tipster program in Recipe 1.15.  It follows in the long tradition (started, possibly, by 1978′s “The C Programming Language”)  of tossing the reader into the deep end with a real program.  
I have been able to make several Android applications run because of this book and a few others that I liked, including Ed Burnette’s Hello Android, 3rd Edition, Reto Meier’s Professional Android 4 Application Development, 3rd Edition, Wei-Meng Lee’s Beginning Android 4 Application Development, and Ian F. Darwin’s Android Cookbook.

A few notes on real programming:

1. You have to use the Google-supplied Eclipse.  Apparently Google got sick of waiting for Eclipse.org to fix bugs.  The Google-supplied Eclipse works better.  

2. Eclipse Run Configuration and Adroid AVDs constantly need to be deleted and recreated.  Some of these problems have been repaired.  But even circa Fall 2013, I find myself repeatedly deleting and recreating the AVD and the Run Configuration in order to get the old AVD to forget.  

3. Creating AVDs to use the Intel and to use Snapshots definitely speeds up development.  These settings should be the default.  Even on the biggest machine available, modest apps take forever in the ARM/no-snapshot default setup.  

4. Be persistent.  Android development is getting easier, but there are hiccups that are not your fault.  Keep working and have good books on hand, including Android Cookbook.

Posted in Uncategorized | Tagged | Leave a comment

Book Review: Webbots, Spiders, and Screen Scrapers, 2nd Edition

Nowadays websites are content management system things where all of the content sits in a database, not in a tree of HTML files. As a website administrator, this all-content-stuck-in-mysql can get frustrating because all of your blogs, products, etc. are in tables. I wanted to look closer at the content inside of a locally running Drupal deployment. Tapping into this content via the mysql command line is mind numbing non-fun. The easy way, I thought, would be to crawl the site with web clients. The last web crawling book I read was Clinton Wong’s excellent Web Client Programming with Perl. Its still a great book after 15 years! Another good book, 10 years on, is Sean Burke’s Perl and LWP. This desire to crawl Drupal content without any effort was in the back of my mind when I saw Webbots, Spiders, and Screen Scrapers, 2nd Edition, A Guide to Developing Internet Agents with PHP/CURL come across the wire. If you know a little PHP, chapters 3, 4, and 6 will take you an hour to read and you’ll be crawling right away. All of the other chapters are short, easy and fun to read. Most chapters are easy to skim or skip. Part 2, “Projects”, is skimmable and skippable. But these cookbook-like projects are easy reads and lots of fun. Easy fun. Easy Fun. Webbots, Spiders, and Screen Scrapers, 2nd Edition is easy fun!

GOOD PARTS:

The book uses some libraries written by the author, like LIB_http.php, that are very simple wrappers around the more extravagant and complicated offerings found in the open source cURL project of Swedish developer Daniel Stenberg. We learn a little about PHP’s integrated web client functions like fopen(), file() and fgets(). I liked the LIB_http.php wrapper because it is very small and easy to read and understand. Using it lets the developer concentrate on web crawling instead of code. It took some nerve for our author to provide a web client library but the gamble pays off keeping us focused on the crawling instead of understanding some behemoth API (as are to be found in many other web client library offerings). Anyone who has navigated the DOM of lousy HTML will appreciate the parsing strategy found in this book. Its more like the unix grep and less like tree-navigation. The HTML parsing priesthood might scoff at the crude parsing found here but I enjoyed it very much as it serves much web crawling needs without drowning in the DOM. These libraries are not super-powerful and won’t serve every need. But they are lots of fun and easy to use. Check out Apache HttpComponents if you want to drown. I was surprise to see that LIB_http.php was still being employed in Chapter 25, “Deployment and Scaling”, to operate a web botnet and also Chapter 28, “Writing Fault-Tolerant Webbots”. If you’re going to run a botnet, I would think that you’ll probably want to get a bigger library, but these simple libraries leverage cURL to get alot done in this book.

The author has clearly done alot of actual, real-world web crawling. He says in the book why he prefers query-string-based session management over cookie-based session management: “…some web developers (myself included) prefer query sessions, as some browsers and proxies restrict the use of cookies and make cookie sessions difficult to implement.” It is also clear that the author has used robots to make online purchases. One wonders whose credit card was on the hook for code quality. It is funny to think of UPS showing up with, oops, 1000 packages one Tuesday morning. The author’s real-world web crawling experience adds something hard to describe, something good.

The section on iMacros was news to me. Unfortunately this chapter relies on code that’s not open source. Sometimes web crawling calls for automating an actual web browser. These macro chapters are wild newfangled fun. If your website target is hard to crawl, pull out a macro on them.

BAD PARTS:

The worst page in the book is page 68, which as 3 problems:
PROBLEM #1
On page 68, during a discussion of web form submission, it says:

“””
Since URL fetch requests are sent in HTTP headers, and since headers are never encrypted, sensitive data should always be transferred with POST methods. POST methods don’t transfer form data in headers, and thus, they may be encrypted. Obviously, this is only important for web pages using encryption.
“””

…but this is incorrect. If the web site is using https, then the whole pipe, including HTTP headers, is encrypted and not available to third-party sniffing. Using GET or POST does not make any difference. This and similar mistakes are repeated in other places including pages 196 and 261.

PROBLEM #2
On page 68, there is a typo on the very last line of Listing 6-6. This line:

$response = http($target=$action, $ref, $method, $data_array, EXCL_HEAD);

…should instead read:

$response = http($action, $ref, $method, $data_array, EXCL_HEAD);

PROBLEM #3
On page 68, last line of Listing 6-6, we use a function called http(). This function is directly callable, but the documentation in LIB_http.php advises:
“””
# http()
# A common routine called by all of the previously described
# functions. You should always use one of the other wrappers for this
# routine and not call it directly.
“””
…we probably would be better off using LIB_http.php’s http_post_form().

Both Oreilly and NoStarch do not have an errata page for reporting errors for this book (that I could find). Fortunately, the book is not overloaded with errors, just one bad page. The writing is generally precise but I did get tired of the repeating assumption that “fetch” is synonymous with “GET”. In http, we use “GET” not “fetch”.

All in all, Michael Schrenk has done us a great favor with his excellent new book. I congratulate him. Webbots, Spiders, and Screen Scrapers, 2nd Edition is easy fun!

Posted in Uncategorized | Leave a comment

Book Review: MySQL Troubleshooting

This is a review of Oreilly’s excellent new book, MySQL Troubleshooting by Sveta Smirnova.
Having read more than half of Oreilly’s MySQL books, I can say that the bar for quality is very high in this genre. The MySQL reference manual is indispensible, but to really learn MySQL, you need to read the Oreillys. These classics in particular, are handy to read:

Learning MySQL

MySQL High Availability

High Performance MySQL, 3rd Edition

MySQL Cookbook, 2nd Edition

SQL Hacks

SQL Cookbook

MySQL in a Nutshell, 2nd Edition

SQL and Relational Theory, 2nd Edition

I expected to breeze through MySQL Troubleshooting but I often found myself short on knowledge and reviewing concepts in other books, especially MySQL High Availability and High Performance MySQL. MySQL Troubleshooting will benefit experienced administrators the most. At times, our author provides seemingly contrived problems and offers paths to detailed and strange minutiae of MySQL troubleshooting. At other times, topics like backup receive short shrift. In the end, though, our author proves to be a worthy guide, a good writer and an expert in this domain.

CHAPTER 1:
We learn how to set turn on query logging:
mysql> SET GLOBAL general_log=’on’;
…which only works for mysql 5.1 and later. For earlier versions, we need a line in /etc/my.cnf like this:
“log=myquery.log”
…and restart mysql. Being stuck with an older MySQL, editing /etc/my.cnf to turn on query logging looks necessary. As with most Oreillys, doing the things in the book is always the way to go. Get yourself a MySQL command-line somewhere and keep it for the whole book.

Long listings of MySQL logging output are to be found in this book. Some of this logging can be painful to read but part of the story here is that MySQL troubleshooting often involves painstaking log analysis. I found myself skimming the logging at times and reading only the explanation found below the logging output. The skimming does not pay–you need to read alot of logging output to understand the concepts driving the troubleshooting effort being illustrated. With a printed book, the reader can highlight or circle the parts of the logging output that really matter, the parts that are referred to in the explanation. But not with a PDF. Perhaps Oreilly should rethink printing 2 pages of logging output without any boldfacing/highlighting that draws attention to the important parts. Oreilly does not usually do this with code and probably shouldn’t. But I don’t find myself skimming code snippets like I do with logging output. Maybe in the new PDF-books-are-everywhere-world Oreilly should markup the pages a little more. I liked the “line-numbering” mechanism Mark Pilgrim uses in his (free at one time) “Dive Into Python” books. In any case, log analysis is performed in this book and you’ll need to squint to grasp the concepts.

I was trying to make a point before I started bellyaching about ebooks: The logging output and its analysis are a great strength of the book. Much of this material will be new even to seasoned admins.

We learn some tricky SQL. From the book:

“MySQL thought we wanted to run a dependent subquery”.

After reading about dependent subqueries and seeing how easy they are to create-with-mistakes, I’m convinced there are alot of dependent-subquery sufferers out there. That said, the SQL query troubleshooting got a bit tiring quickly. If you want to learn SQL, pick up OReilly’s excellent “Learning SQL” or Chapters 5,6 & 7 of “Learning MySQL” or Head First SQL or (if you’ve got 6 months) C.J. Date’s Relational Theory book.

Unfortunately, starting with Chapter 1, proficiency with the basics of master-slave replication setup and operation is assumed. We were warned about this in the Preface. (Another thing we remember during Chapter 1 is that we were also warned that the programming examples will be based mostly on the C API). Fear not–replication is easy. Reading the first 43 pages of MySQL High Availability, or Chapter 10, Replication, in “High Performance MySQL, 3rd Edition” will have all the basics you will need to follow along with most of the replication troubleshooting covered in this book.

CHAPTER 2:
Chapter 2 shows some table and row locking problems very clearly. We manufacture a locked table and a locked row and look at what “SHOW PROCESSLIST;” thinks about it. You’ll need a mysql client command line to really enjoy the fun. Run “SHOW ENGINE INNODB STATUS”. I also took a very long time pondering this quote about how performance relates to INSERTS with indexes and locks, from Chapter 2:

“when row locks are in use, indexes can increase overall application performance, especially when an index is unique, because while updating such an indexed field, insert would not block access to the whole table.”

I still don’t understand it. I have the PDF version, and right now I want to circle the confusing quote and fold the corner of the page so that I can return later. The nice thing about the PDF version is that the book is packed with links to context-specific help. When I click on them, it takes me right to the content that I want to read more about. I took many trips to the MySQL Reference Manual while reading this book. With a hardcopy book, I would have taken zero trips to the web. The PDF being able to search the text is also nice–but some of the text in my PDF was not searchable–that is not cool.

If you think you know everything, think again! In this book we learn that these statements:

mysql> CREATE TABLE t1(f1 INT) ENGINE=InnoDB;
mysql> SELECT * FROM t1;
mysql> BEGIN;
mysql> INSERT INTO t1 VALUES(100);
mysql> CREATE TABLE t2 LIKE t1;
mysql> INSERT INTO t1 VALUES(200);
mysql> ROLLBACK;

…result in t1 having one row with f1=100. That’s because “CREATE TABLE” implies a COMMIT.

How about this hack:
mysql -A test
mysql> CREATE TABLE innodb_monitor(f1 INT) ENGINE=InnoDB;
…which mysql notices and starts outputting to the error logfile. This allows you to find the query that is holding the lock (and the query waiting for the lock) by lining up timestamp information from the slow-query-log. Nice.

CHAPTER 3:
Chapter 3, Effects of Server Options, excels at providing bite-sized troubleshooting. We learn that its possible to get a MyISAM table from this query:

CREATE TABLE t1(f1 INT) ENGINE=InnoDB;

Ouch! The discussion on max_allowed_packet should have run longer. The author tells us that: “In my day-to-day job, I see many users who are affected by ignoring the value of max_allowed_packet”. She can add me to the list of many users. We get a poor explanation of how to address this problem:

“If you start getting syntax errors for valid queries, check whether their size exceeds max_allowed_packet.”

The Mysql Reference manual tells us this:

“The largest possible packet that can be transmitted to or from a MySQL 5.1 server or client is 1GB.”

…but if query-size and packet-size are the same, then how would anyone insert a large 2Gig blob value? I would have appreciated more depth on troubleshooting problems with max_allowed_packet. Google returns 853,000 results for searches on “max_allowed_packet”, so we are not alone.

At about halfway through the book, I had a disturbing revelation. This quote pushed me over the edge:

“Both the –no-defaults and –defaults-file options must be specified as the first options passed to mysqld. Otherwise, the server will not recognize them”

…that was one too many mysql gotchas for my stomach. This book makes mysql look bad. But better to be bad-and-known-about than bad-and-not-known-about.

Chapter 3 is packed with awesome server option explanations and tips…my favorite part are these queries:

SELECT (@@query_cache_size + @@innodb_additional_mem_pool_size + @@innodb_buffer_pool_size + @@innodb_log_buffer_size + @@key_buffer_size)/(1024*1024);
SELECT @@max_connections * (@@global.net_buffer_length + @@thread_stack + @@global.query_prealloc_size + @@binlog_cache_size) / (1024 * 1024)

…which tell you, in megabytes, how much memory the running mysqld needs to allocate these buffers. The entire memory discussion is excellent and important now that terabyte-sized machines are available.

CHAPTER 4:

Chapter 4, MySQL’s Environment, is mostly about performance and is packed with great tips. Topics include CPUs, memory and disk. No surprises here–disk and memory help, CPUs don’t. I was surprised to read on page 151 that:

“Clients almost always connect to the MySQL server over the network”

…perhaps so, but it would be nice to know how or why the author believes that. Whether and why to run MySQL locally or on another box is an important decision that deserves more attention than we get here. Perhaps you’ll be better off running mysqld locally with the database running on dedicated disks for the datafile, the log file and the binlog.

The slowest part of Chapter 4 is when we learn about these latencies associated with a typical query on a typical server:

“The client sends a command to the server that takes a half of RTT (network round-trip-time).
The WHERE clause of the UPDATE is executed, and mysqld reads the disk.
mysqld does an fsync call to prepare for the transaction because autocommit is on.
mysqld does an fsync call to write into a binary logfile.
mysqld does an fsync call to commit changes.
The client receives the result from the server, which is another aspect of RTT.”

CHAPTER 5:

Chapter 5, Troubleshooting Replication, is again packed with great tips, lucid explanation and links to other documentation and software. Some of the treatment is too brisk, especially the explanation of “SHOW SLAVE STATUS” and multi-master troubleshooting. There is an excellent example of using mysqlbinlog to check the master’s binlog and the slave’s relay log for corruption. Brushing up on replication concepts is a good idea before reading this chapter–Chapters 2 and 3 of “MySQL High Availability” is a good re-read for this purpose, especially page 87′s mysqlbinlog demonstration.

CHAPTER 6:
Chapter 6, Troubleshooting Techniques and Tools, starts with the author explaining why simple tools will be preferred in this chapter. How nice. No explaining necessary. The simplicity stops when the author suggests three ways of addressing slow queries, one of which is:

“write a MySQL plug-in for auditing purposes”

…How flattering of her to think I’m talented enough to write and compile a mysql plugin to deal with a slow query! Seriously, though, by following her link to http://dev.mysql.com/doc/refman/5.5/en/writing-audit-plugins.html, writing such a plugin looks easy.

The third solution, write a scriptable proxy, looks like lots of fun too. We get a dip into Lua programming.

These solutions are very cool. But aren’t slow queries always caused by a new query of a big table on a column without an index? Troubleshooting this simple problem could have used more than the advice: “add an index” that it gets in this chapter.

We learn how to use “mysql –column-type-info test”, how to (safely) operate on copies of tables, how to use Giuseppe Maxia’s Mysql sandbox tool (a totally wild and crazy thing that actually works), how to spit out and analyze a core file, how to extract goodies from INFORMATION_SCHEMA, mysqlslap, sysbench, Gypsy, mysql-test-run, and more.

The book crosses into the absurd when our author commands us:

“At some point, of course, you have to dive into the MySQL source code itself.”

Holy cow! Actually, dear author, no, I don’t need to dive into source code. Your book will do just fine with these eyeballs.

The InnoDB Monitor section was very tough reading for those of us low on InnoDB details like the least-recently-used algorithm. Even so, playing around is easy to do by simply creating tables innodb_monitor, innodb_lock_monitor, innodb_table_monitor and innodb_tablespace_monitor. Note to self: come back later.

I always thought mtop and Mysql Workbench were pretty cool, but they get little coverage here.

The climax of the book occurs on page 206:

“General Steps to Take in Troubleshooting”

It takes a leader with some nerve to attempt such a section title with MySQL. The advice is good.

The section on the MySQL Test Framework is good but my local CentOS 5.5 linux box did not have the mtr command. I installed it with “yum install mysql-test.i386″ (or yum install mysql-test.x86_64). The book calls the command “mtr”. But on my linux box, mtr is a little network program called “my traceroute”. The mysql-test binary is called “mysqltest” on CentOS5.

CHAPTER 7:

Chapter 7, Best Practices, wanted to be longer. It is surprisingly basic and easy reading after Chapter 6 when we were writing Lua and analyzing core dumps. Table 7-1 compares backup tools nicely.

GENERAL NOTES:
The book misses opportunities to share wisdom in too many places. For example, when discussing the issue of troubleshooting MySQL permission problems, the author offers detailed information about how to track down the problem. But the wisdom of MySQL permission problems, I’ve found, is to avoid using the power at all. MySQL has always come with the ability to extremely fine tune permissions–admins can enforce rules such that a certain user at a certain host can access a certain column in a certain table in a certain database and nothing else. But elaborate MySQL permission settings is rarely necessary and always asking for trouble. Most shops will only need to carefully grant proper access to apache and a few developers. MySQL security is more about simplicity, backups and documentation. But when the author does try to pass along wisdom, she does so very well, like in the “Haste Makes Waste” section, where we are told to start mysqld with the –no-defaults option and then set each of the local custom options, one at a time, measuring performance with each setting change. Another piece of advice we all ignore is to ignore warnings. Repeatedly, when we see a warning in the book, we track it down. Tolerate zero warnings is the wisdom.

The book could have used a few more “refresher” paragraphs. A quick refresher about replication would have helped.

MySQL Troubleshooting is an excellent book. I congratulate the author and Oreilly for sharing it with us.

Posted in Uncategorized | Leave a comment

Book Review: Environmental Monitoring with Arduino Building Simple Devices to Collect Data About the World Around Us

This is a review of the new book “Environmental Monitoring with Arduino Building Simple Devices to Collect Data About the World Around Us” by Emily Gertz and Patrick Di Justo. I am an experienced programmer but very new to Arduino. To get started with Arduino, the path I took is not a bad one, in this order:

  1. Install the Arduino software for Mac, Windows or Linux.
  2. Get the Getting Started with Arduino kit V3.0 from makershed.com
  3. Read Massimo Banzi’s excellent book “Getting Started with Arduino, 2nd Edition”.



Some programming experience will make Arduino-ing more fun.


Arduino programs, called Sketches, are written in a Java-like language (called Processing) and are very basic. Basically, you write and call two parts, setup() and loop(). setup() is where you specify where the input is coming from (a pin) and where the output is going (an LED light). loop() runs forever. That’s it for programming. After getting your computer connected to your Arduino hardware properly and writing your program in the Arduino IDE you downloaded (see step #1 above), you press the “Verify” button and then the “Upload to I/O board” button. Seeing something blink can be very satisfying.


Netduino is coming out and the major difference (from Arduino) is that its programs are written in C#. So if you are a Java or .NET programmer, you might instead get Oreilly’s book Getting Started with Netduino by Chris Walker (circa March 2012). And you might instead get hardware better suited for Netduino development.


Back to Arduino. Step #4 in learning Arduino is to go in any of one hundred directions. You should see what Oreilly has and makershed.com if you’re low on ideas. That’s where I picked up “Environmental Monitoring with Arduino Building Simple Devices to Collect Data About the World Around Us”.


All Arduino books and hardware require activity. Don’t read this book without doing the things in the book. You need to have some hardware! Ebay might be a good place to get stuff but I am too newbie to shop for Arduino auctions. Two books and a starter kit is $100.


I am using the PDF version of the book and its pictures are beautiful. There are errors, but I haven’t found any bad ones yet, and the errata page is still empty.


The projects in the book achieve noise detection, electromagnetic interference detection, water conductivity measurement, internet connectivity, nuclear radiation detection and lots of output trickery.


The Preface of the book says that “environmental monitoring” requires “complicated and expensive equipment” typically used by “scientific experts at government agencies, universities, and corporations” occasionally compromised by “their own institutional agendas”. The message here is that we can do it ourselves cheaper and better with a little help from Arduino. Maybe so. And maybe citizen monitoring has a bright future. But the environmental monitoring projects in this book are fun and can stand on their own as simple fun.


The “How to use this book” section in the beginning offers classic Arduino development advice. The whole Arduino stack does breakdown and fail too often. The thing you want to happen simply doesn’t. Follow the advice found here to minimize failure.


One problem with this book is that you need certain hardware that you don’t have right now to complete the project. A great strength of the book is that each section lists the parts you need. For example, Chapter 3 requires a thing called a “4Char display”. The “Parts” section lists a part number: “SparkFun sku COM-09765A”. This is great help. It would have been better if there was a list at the beginning of the book with all the parts needed to do all of the projects in the book, some advice on what they will cost and where to get them. The book does suggest these four providers of Arduino hardware:



If I did this over again, I would have bought these parts before reading the book. Even better would be if makershed.com provided a kit made just for this book (like they do for Banzi’s book). Because as it was, I was skimming too many chapters knowing that I needed hardware and wondering whether I should bother the UPS guy or not. Usually “not” prevailed, call me lazy.


There are dozens of universities that have both a Computer Science Department and some kind of Environmental Studies Department. The CompSci undergrads won’t be deploying hardware in the field, but usually they are studying Java and so can write Processing. The Environmental Studies kids won’t be writing code but they will deploy and troubleshoot sensor hardware. A few dozen kids from these Departments equipped with a few dozen boxes of Arduino hardware and a few dozen Arduino books all mixed up might generate a nice blob of data that could be taken to the Statistics Department.




Emily Gertz and Patrick Di Justo have written a fine book.

Posted in Uncategorized | Leave a comment

Book Review: Head First iPhone and iPad Development, 2nd Edition

This is a review of Oreilly’s excellent “Head First
iPhone and iPad Development, 2nd Edition”.

I own about two dozen different books about C and
C++ programming. I have spent several hundred hours
learning and doing C programming. I find it puzzling
that almost every college and university in America
seems to think that learning C or C++ is a 3-credit
endeavour. Learning C (without piling on any of C++)
is 9 college-level credits, in my opinion. C, we
are told, is very simple. Although it is true that
you can fit the entire language inside your head,
C is not simple. J Everyone should still start with
“The C Programming Language” by Brian Kernighan
and Dennis Ritchie. That book is not an easy read,
but always a great investment. It only scratches
the surface of topics like the stack, the heap,
and memory management. I remember reading Frantisek
Franek’s “Memory as a Programming Concept in C and
C++” and feeling betrayed–why hadn’t anyone told
me these things before? Ten years ago, it was sad
that universities were offering freshman classes like
“Computer Science 201: C++ Programming”. A 3-credit
course never has and never will make a C++ programmer
(or a C programmer for that matter). It is sad today
that Universities are offering Freshman classes like
“Computer Science 201: Java Programming”. Java and
C++ both sit upon C and should be taught after C,
especially in Electrical Engineering curricula.
If Computer Science departments want to skip C and
move directly to Java, as many currently do, then
they should commit to measuring their performance.
Find yourself any recent graduate of a Java-based
CompSci department and see if they “know” Java.
They don’t!

This context is why I was alarmed to see Oreilly’s
“Head First iPhone and iPad Development, 2nd Edition”.
This unlikely title is surprising to me because
there are two things that I already know about iPhone
programming:

1. It requires knowledge of Objective-C.
2. It requires knowledge of Apple’s frameworks.

Each one of these problems requires hundreds of hours
of effort to reach basic proficiency. Objective-C is
a superset of C and thus is very hard.

When I started reading “Head First iPhone and iPad
Development, 2nd Edition”, I knew almost nothing
about what Objective-C adds to C or about any
Apple frameworks. And so “Head First iPhone and
iPad Development, 2nd Edition” seemed like a good
book for me to read. I loved it. Undaunted by the
challenge, Dan and Tracey Pilone have delivered a
masterpiece of efficient learning. This book will be
useful to anyone involved at any level of iPhone/iPad
development. Even managers! Everyone reading the
book, like all Head First books, owes it to themselves
to follow along as advised. Do not resist! You must
have an Intel Mac nearby where you are reading.
You must register, get the SDK, Xcode, etc. Even
without expertise in Objective-C or Apple frameworks,
you can easily follow along with this book.

To learn iPhone/iPad development and build useful
apps, you will eventually need to learn Objective-C
and the frameworks that support Apple programming.
This is a 1000-hour effort (that’s 4 hours every
weekday for an entire year). The book does not make
that clear. As I was getting through the book,
I took a sideways, concurrently, to pull in more
Objective-C. To do so, I was using Oreilly’s “Cocoa
and Objective-C: Up and Running” and “Objective-C
Pocket Reference” hardcopies. Many iPhone/iPad
developers are using Apple’s online learning tools
and iTunesU and movies to get their skills.

If I were new to iPhone/iPad development, I would be
prepared to pay the 1000-hour minimum. And I would
start by spending 40 or 50 hours getting through
Oreilly’s excellent “Head First iPhone and iPad
Development, 2nd Edition”.

I review for the O'Reilly Blogger Review Program

Posted in Uncategorized | Leave a comment

Installing Drupal 7 on Mac OSX 10.6 Snow Leopard

Installing Drupal on a Mac using MacPorts:
First steps:
1. Install Appleʼs “Development Tools” which youʼll find on the “Applications” DVDrom
that came with your Apple machine.
2. Run “Software Update” in “System Preferences” on your Mac to get the updates from
Apple for the Development Tools you just installed.
3. Download and install Macports from macports.org. This is a .dmg file that you
double-click on and go–easy.
To install Drupal7 on your Apple, we are going to use everything Macports. Everything
will be done on the command-line using Macʼs Terminal program (which youʼll find in
“Applications–>Utilities–>Terminal”). Many of the commands we type will require
superuser permissions and so weʼll be prefixing our commands with “sudo” and
sometimes it will make you type a password for a user with admin-level permissions on
the Mac.
Instead of macports, you can also use the default Apache that comes with every Apple,
and install a double-clickable mysql-server .dmg file from mysql.com and manually
install Drupal7 from drupal.org. Or you could instead install Drupal7 using the MAMP
method. Macports is a good option for two important reasons:
A. Macports is great software compiled for your Mac.
B. It is often used by experts and therefore the online help youʼll find is usually good.
4. Update Macports..in Terminal type this and wait a few hours:
sudo port selfupdate
5. Install apache2, php, mysql client, mysql-server, drupal7:
sudo port install apache2
sudo port install mysql5
sudo port install mysql5-server
sudo port install php52
cd /opt/local/apache2/modules
sudo /opt/local/apache2/bin/apxs -a -e -n “php5″ libphp5.so
sudo port install drupal7
sudo port installed
sudo port list
sudo port list | grep agick
6. Setup mysql-server:
sudo port load mysql5-server
cd /opt/local/bin
sudo -u _mysql mysql_install_db5
cd /opt/local/; sudo /opt/local/lib/mysql5/bin/mysqld_safe &
/opt/local/lib/mysql5/bin/mysqladmin -u root password ʻs3kr3tʼ
cd /opt/local/bin/
mysql5 -u root -p
mysql> quit
7. Copy the Drupal files to apache2ʼs document directory:
cd /opt/local/www/data/drupal7/
sudo cp -a ./ /opt/local/apache2/htdocs/
sudo cp /opt/local/etc/php5/php.ini-dist /opt/local/etc/php5/php.ini
8. Restart apache2 and make changes to fix apache2:
sudo /opt/local/apache2/bin/apachectl -k restart
links http://localhost/ (gives you “It works” from index.html).
links http://localhost/index.php (gives you php source code)
….at this point two problems still exist:
1. Apache is returning index.html as default document, not index.php
2. Apache is not parsing PHP code, just returning the source code as text.
…to fix the index.html-vs-index.php problem we change DirectoryIndex ordering.
…to fix the PHP not parsing problem, add an “AddType PHP” directive to httpd.conf
Fix both problems like this:
******************************************************
sudo cp /opt/local/apache2/conf//httpd.conf /opt/local/apache2/conf/httpd.conf.ORIG
sudo vi /opt/local/apache2/conf//httpd.conf
sudo /opt/local/apache2/bin/apachectl -k restart
sudo cp /opt/local/apache2/conf//httpd.conf /opt/local/apache2/conf/httpd.conf.070611-
WORKS
links http://localhost/index.php (gives you Drupal page YAY!).
******************************************************
..if you donʼt know vi, try using nano or pico which are easier text editors to use.
..during our vi editing session, we commented-out one line and added two lines:
#DirectoryIndex index.html
DirectoryIndex index.php index.html
AddType application/x-httpd-php .php
…and now we have a working setup and backup httpd.conf. Notice that index.php
comes before index.html…this causes the php file to go out by default if both are
in the directory being requested by the user. The AddType line makes PHP work (that is,
it causes apache2 to execute the php source code instead of returning verbatim code to
the user).
9. One file that came with Drupal and is worth looking over is /.htaccess. Drupal follows
(or should be following) the directives in this file.
10. At this point, Drupal is working but is not setup. There are a few additional pieces of
administration necessary. What still needs to be fixed is not obvious. But because
Drupalʼs php code can execute, letʼs run it and see that it offers helpful
troubleshooting. When it complains about a problem, we will address and fix it until
it doesnʼt complain. At this point, Step #10 is the last “step”. From now on, we will
be clicking on links and filling out forms in our web browser and also running
command-line operations. These are easier to explain by discussion instead of
listing discrete steps.
On our fresh new site, when we visit http://localhost/ we get index.php, but it
in turn sends us to install.php (presumably because Drupal knows it hasn’t
been setup yet). The questions on the install.php page are:
———————–
Choose profile(active)
Choose language
Verify requirements
Set up database
Install profile
Configure site
Finished
———————–
…we should document our deployment choices. The first choice (profile)
asks us if we want “standard” or “minimal” and lets us click a button that
says “Save and Continue”.
…Now is a good time to read up on these choices so that we make good
choices…and then we’ll document them so we know which choices were made.
Continuing along, we are now at this Drupal location in our web browser:

http://localhost/install.php

…and we are making choices:
Choose Profile—–”Standard”
Choose Language—-”English”
Verify Requirements—NOT GOOD! These problems exist:
1. Database Support _NO
2. File System:”The directory sites/default/files does not exist”
3. Settings File:”./sites/default/settings.php file does not exist.”
…this is not too bad. To fix problems 2 and 3, we will make the “files” directory and will
copy in a default “settings.php” file to the proper location. We will also make the whole
directory owned by the apache2 user so that it has the permissions necessary to create
new files (so that users can upload files…and so that Drupal admins can add Drupal
modules). Run these commands in your command-line terminal:
******************************************************
cd /opt/local/apache2/htdocs/
ps -ef | egrep “PID|http”
cat /etc/passwd | grep 70
sudo mkdir sites/default/files
sudo cp sites/default/default.settings.php sites/default/settings.php
sudo chown -R _www ./
sudo /opt/local/apache2/bin/apachectl -k restart
******************************************************
…notice that we check (again) and see that apache is running as userid 70,
which on this MacOSX10.6 box’s /etc/passwd file indicates user “_www”. We
thus change the ownership of the htdocs directory recursively (all files/dirs below)
to new owner “_www”. Changing ownership is the only permissions change that we
should need to make because Drupal ships the files with the proper permissions by
default–for example–the “files” and “modules” and “themes” directories all ship owner-
writable so that users can upload files and admins can install modules and themes.
We also make a backup copy of the settings file and restart apache.
…after performing these steps and reloading our web browser on this page:

http://localhost/install.php

…both problems #2 and #3 went away.
…the first problem, “No Database Support”, was fixed like this:
******************************************************
sudo port uninstall php52
sudo port install php52 +apache2 +mysql5 +pear
cd /opt/local/apache2/modules
sudo /opt/local/apache2/bin/apxs -a -e -n “php5″ libphp5.so
sudo /opt/local/apache2/bin/apachectl -k restart
******************************************************
…first we reinstalled php52 but added some extra “extensions”. The
macports php52 comes with several extensions, including:
curl, freetype, jpeg, libpng, libmcrypt, libxml2, libxslt, mhash, and tiff
…but we’re also going to want apache2, mysql5 and pear.
…after running the commands shown and reloading this Drupal setup page:

http://localhost/install.php

…the “Database Support” problem was fixed.
at this point in our Drupal7 setup, the “Verify Requirements” page reports “No
problems”–yay.
…continuing to go through the steps on the Drupal7 setup page at:

http://localhost/install.php

…we are now on the “Database Configuration” page. It wants to know:
1. Database name (and this database must already exist).
2. Database username
3. Database password
…this page won’t be successfully completed until we create the database
and a user/pass that has permissions on it. These steps were run to set that
up:
******************************************************
cd /opt/local/bin/
./mysql5 -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.1.57 Source distribution
Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL v2 license
Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.
mysql> create database drupalseven;
Query OK, 1 row affected (0.00 sec)
mysql> GRANT ALL ON drupalseven.* TO ‘drupaluser’@’localhost’ IDENTIFIED BY
‘QW1ET’;
Query OK, 0 rows affected (0.00 sec)
mysql> quit
Bye
******************************************************
…we may need to flush/reload/??? to get mysql to re-read its permissions.
Or maybe they work out of the box. Restarting mysqld does not appear to be
easy on this macports/osx10.6 box. To restart mysql, these steps were
run:
******************************************************
cd /opt/local/bin/
./mysql5 -u root -p
ps -ef | grep mysql (OUTPUT SHOWS mysqld_safe has PID 2424)
sudo kill -HUP 2424
./mysql5 -u drupaluser -p (WORKS)
******************************************************
…and with that we have a database named “drupalseven” and a user named
“drupaluser” whose password is “QW1ET”. User “drupaluser” was granted
all permissions on every table in the “drupalseven” database. We can now
continue by filling in these items on the “Database Configuration” portion
of the Drupal7 setup at:

http://localhost/install.php

DBNAME: drupalseven
DBUSER: drupaluser
DBPASS: QW1ET
…after entering this information, and clicking to continue, the
next page looked good, there was a status bar, and 28 of 28 things
got done…most of the text that went by said things about installing
modules, like this: “Installing the Toolbar Module”.
…The next page of the Drupal7 setup page at:

http://localhost/install.php

…is called “Configure Site” and asks for these things to be entered:
Site Name:
Site Email:
Site Maintenance Account:
—-user
—-email
—-pass
—-re-enter pass
Server Settings:
Default Country:
…at this point a good question is how the mail is going to work. If I paste
in a gmail address, how will I know if my machine and network will let it all
out. OSX10.6 does not ship with a running mail server because this
command just hangs:
******************************************************
mail -s “Testing RU There” `whoami`@`hostname`
telnet 127.0.0.1 25
Trying 127.0.0.1…
telnet: connect to address 127.0.0.1: Connection refused
telnet: Unable to connect to remote host
******************************************************
…try adding a gmail/yahoo and see if it lets you continue. Otherwise, these steps might
work:
******************************************************
sudo launchctl start org.postfix.master
telnet 127.0.0.1 25
sudo launchctl start org.postfix.master
telnet 127.0.0.1 25
mail localusername
less /var/mail/localusername
hostname
******************************************************
…whatever you get from the hostname command, write it down. Otherwise, you can
disable all of your networking (airport off, ethernet disabled, etc.) and type the
“hostname” command again to get a hostname like “Local-Usernames-Imac.local”.
When entering the Drupal-setup-email information, try one of these possibly-valid local
email addresses:
localuser@hostname.pitt.edu
localuser@Local-Usernames-Imac.local
…with any luck Drupal7 says:
“Drupal installation complete”
YAY!
…it prompts me with a link to “visit your new site”, that goes here:

http://localhost/

The Drupal7 setup will annoyingly not complete unless you can get past the Drupal7-
email-setup page. And the only way to do that is to enter a valid email address. This is
a drag because:
1. Your network might not let Drupal send out email (to yahoo for example).
2. Getting your local Mac to properly run an email server is not so simple especially
because your hostname is apple-unusual. Further, even if you do get it working
during drupal setup, most apple-hostnames change often. Your Mac laptopʼs
hostname this morning might have been “unpaiduser112@dhcp-forbes-
Pgh.starbucks.com” but later today it will be different and your Drupal7 email settings
willl be wrong.
3. Drupal7 disallows real and useable email addresses including these:
localuser@127.0.0.1
localuser@localhost
A good strategy is to just get past this portion of the setup process and worry about
fixing it later (or just look at logging information online instead of reading emails sent to
you by Drupal).
At this point, Drupal7 setup is complete. Very little will be necessary on the command-
line from now on. (Almost) everything Drupal is done via its web interface. Hence, this
is a good time to backup your Drupal site. Backing up Drupal can be done in many
ways including by using Drupalʼs “Backup & Migrate” Drupal module. Instead, we will
take a backup of the databases and the htdocs directory. All changes in Drupal will
manifest themselves (on this machine) at /opt/local/apache2/htdocs and /opt/local/var/
db/mysql5/. Notice that the htdocs directory might contain more web content than just
your Drupal stuff. Also notice that the mysql5 directory contains things other than your
Drupal databases–it contains all of the mysql databases. After taking this backup, we
will have an archive that can be used to restore our Drupal deployment to its exact state
at the time the backup was taken. Run these commands to take a backup:
******************************************************
ps -ef | egrep “PID|mysql”
sudo kill -9 560 (560 is the mysqld_safe PID)
sudo kill -9 561 (561 is the mysqld PID)
cd /opt/local/apache2/
sudo ./bin/apachectl stop
sudo tar -czf ./htdocsBackup070611.tar.gz ./htdocs/
cd /opt/local/var/db/
sudo tar -czf ./mysql5Backup070611.tar.gz ./mysql5/
******************************************************
…those two .tar.gz files contain our entire Drupal7 deployment so far and can be used to
restore the site to its current condition. To restart mysql and apache2, run these
commands:
******************************************************
cd /opt/local/; sudo /opt/local/lib/mysql5/bin/mysqld_safe &
cd /opt/local/apache2/
sudo ./bin/apachectl start
******************************************************
…your Drupal site should be running normally at http://localhost/.
When restoring a backup, typically you want to preserve the existing site. We will do
that. To restore your backups, run these commands:
******************************************************
ps -ef | egrep “PID|mysql”
sudo kill -9 560 (544 is the mysqld_safe PID)
sudo kill -9 561 (578 is the mysqld PID)
cd /opt/local/apache2/
sudo ./bin/apachectl stop
cd /opt/local/apache2/
mv htdocs htdocs_BrokenSite_070711
sudo tar -tzf ./htdocsBackup070611.tar.gz
sudo tar -xzf ./htdocsBackup070611.tar.gz
cd /opt/local/var/db/
mv mysql5 mysql5_BrokenSite_070711
sudo tar -tzf ./mysql5Backup070611.tar.gz
sudo tar -xzf ./mysql5Backup070611.tar.gz
cd /opt/local/; sudo /opt/local/lib/mysql5/bin/mysqld_safe &
cd /opt/local/apache2/
sudo ./bin/apachectl start
******************************************************
…the “mv” command is the rename command. Both times we ran it to preserve the
existing site by renaming the folder something besides htdocs or mysql5. When we run
the tar commands, we will be overwriting anything currently at htdocs or mysql5. tar
was run with -tzf which lists the files in the archives and -xzf which extracts the files from
the archives. Its always a good idea to run tar with the -t option first to make sure the
archive contains what you expect it does. If your site stays small, these backups wonʼt
be big and can be taken often. While under development, you can take a backup, then
do some risky/unknown Drupal development without worry. If things go badly, you can
just untar your trusty backup and start that risky part over again. Because Drupal and
mysql are relatively platform-independent, you should be able to take these two
archives to another machine (even a Linux machine) and untar them there and get your
site running elsewhere without too much trouble. Another reason to take backups is to
track development with documentation. Because most Drupal “administration” takes
place via its web interface, it is very easy to lose track of what administration has been
performed. Documenting Drupal administration will get very tedious because constantly
writing down what you are doing will result in dense, unreadable prose. A better way to
document Drupal administration might be like this:
1. Take a backup of Drupal.
2. Perform Drupal administration…module installation, etc. until you get it right, jot down
notes.
3. Restore backup taken in Step 1.
4. Redo successful steps taken in Step 2 (minus all the false leads/mistakes).
5. Write down the successful operations performed in Step 4.
…in this way, you can keep track of your Drupal site setup without having to remember
anything.
Backups like these are too easy and too useful to take that it never makes sense to not
get in the habit of doing it. If you only want to capture the Drupal mysql database (and
not the whole entire mysql directory), run instead this command sequence:
******************************************************
cd /opt/local/var/db/mysql5/
sudo tar -czf ./mysql5_drupalsevenBackup070611.tar.gz ./drupalseven
******************************************************

Posted in Uncategorized | Leave a comment