Debugging a strange MacOS X printing problem

At some point in time, something changed on my system that resulted in my printer no longer printing. (Cue a relevant scene from Office Space here…) A quick Googling of relevant keywords didn’t turn up anyone else complaining about what I was observing, so I did what any lazy person would do: I found another way to print what I needed to print, and forgot all about it.

Now, several months later, the problem still persists, and while I found a suitable workaround (use the “Generic PostScript printer” driver instead of the Lexmark one), the hardcore geek in me felt it necessary to struggle against the injustice of this whole “it doesn’t work” thing. It should work, damn it.

First, here’s what the most obvious symptom looks like:

/Library/Printers/Lexmark/filter/pstopsprinter1 failed

Along with this will be an entry in the system log which you can see in Console.app:

1/25/13 12:19:02.275 PM ReportCrash: Saved crash report for pstopsprinter1[78490] version ??? (???) to /Library/Logs/DiagnosticReports/pstopsprinter1_2013-01-25-121902_localhost.crash

If you look inside the crash report, you’ll see:

*** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '-[__NSCFArray length]: unrecognized selector sent to instance 0x1006036b0'

Great. Just great. So, we fire up our handy-dandy debugger, and what is the object it’s choking on?

(gdb) po 0x1006036b0
<__NSCFArray 0x1006036b0>
[redacted],
panoptic.com
)

Aww, damn. I recognize what that is. Those are the domains defined in my /etc/resolv.conf‘s “search” parameter.

Here’s the whole backtrace:

(gdb) bt
#0  0x00007fff93b09ce2 in __pthread_kill ()
#1  0x00007fff9392a7d2 in pthread_kill ()
#2  0x00007fff9391ba7a in abort ()
#3  0x00007fff8ce037bc in abort_message ()
#4  0x00007fff8ce00fcf in default_terminate ()
#5  0x00007fff8d9931b9 in _objc_terminate ()
#6  0x00007fff8ce01001 in safe_handler_caller ()
#7  0x00007fff8ce0105c in std::terminate ()
#8  0x00007fff8ce02152 in __cxa_throw ()
#9  0x00007fff8d992e7a in objc_exception_throw ()
#10 0x00007fff954c81be in -[NSObject doesNotRecognizeSelector:] ()
#11 0x00007fff95428e23 in ___forwarding___ ()
#12 0x00007fff95428c38 in __forwarding_prep_0___ ()
#13 0x00007fff9539f426 in CFStringGetLength ()
#14 0x00007fff953affac in CFStringFindWithOptionsAndLocale ()
#15 0x00007fff953b67da in CFStringHasSuffix ()
#16 0x00007fff953ed7f6 in -[__NSCFString hasSuffix:] ()
#17 0x00007fff8e1ab6da in __-[NSHost resolveCurrentHostWithHandler:]_block_invoke_7 ()
#18 0x00007fff90497c75 in _dispatch_barrier_sync_f_invoke ()
#19 0x00007fff8e1a9b5d in -[NSHost resolveCurrentHostWithHandler:] ()
#20 0x00007fff8e1aa5a7 in __-[NSHost resolve:]_block_invoke_1 ()
#21 0x00007fff90495a82 in _dispatch_call_block_and_release ()
#22 0x00007fff904972d2 in _dispatch_queue_drain ()
#23 0x00007fff9049712e in _dispatch_queue_invoke ()
#24 0x00007fff90496928 in _dispatch_worker_thread2 ()
#25 0x00007fff9392a3da in _pthread_wqthread ()
#26 0x00007fff9392bb85 in start_wqthread ()

Well, let’s see what it was trying to resolve, maybe that’ll give us a clue:

(gdb) frame 20
#20 0x00007fff8e1aa5a7 in __-[NSHost resolve:]_block_invoke_1 ()
(gdb) po $rdi
[redacted].panoptic.com

Yup, that would be the DNS name of my machine, based on the reverse DNS of my IP address on my local private network. However, it appears that forward DNS for that FQDN isn’t resolvable, due to a recent change in my DNS setup. Let’s fix this, and correct the DNS so that the FQDN resolves, and test again.

Still fails. Damn, not going to be that easy, huh?

(gdb) frame 13
#13 0x00007fff9539f426 in CFStringGetLength ()
(gdb) info frame
Stack level 13, frame at 0x1003d72c0:
 rip = 0x7fff9539f426 in CFStringGetLength; saved rip 0x7fff953affac
 called by frame at 0x1003d7770, caller of frame at 0x1003d72a0
 Arglist at 0x1003d72b8, args: 
 Locals at 0x1003d72b8, Previous frame's sp is 0x1003d72c0
 Saved registers:
  rbx at 0x1003d72a8, rbp at 0x1003d72b0, rip at 0x1003d72b8
(gdb) po $rbx
<__NSCFArray 0x1006036b0>(
[redacted],
panoptic.com
)

And, here it is. CFStringGetLength is expecting a CFStringRef and instead getting handed a NSCFArray, and is blowing up because CFStringGetLength trying to [str length] it.

I wish I knew someone who was doing OSX development at Apple to pass this bug along to …

Updated: I filed this as a bug in Apple’s Radar bug reporting system. It was assigned bug ID #13089829.

iTunes 11 is a total failure

ITunes 11

I just accepted the upgrade to iTunes 11 this morning, and am so incredibly disappointed. Can I downgrade back to iTunes 10.7? Thankfully, the iPhone wiki has an excellent list of Apple’s direct download links for past versions! Later, I’ll figure out how to uninstall this abomination and downgrade.

Removal of iTunes DJ is a huge mistake: it was the only feature of iTunes that actually kept me using it as a media player instead of using a non-Apple one. The “Up Next” functionality that Apple replaced it with is NOT a suitable replacement for iTunes DJ. The beauty of the iTunes DJ implementation was that it acted like a special Smart Playlist, which you could see both currently queued and previously played songs in one view, and easily drag-and-drop songs in the queue to rearrange their play order along with dragging new songs into the queue to play them in the order you want. While you can click-and-drag to reorder in “Up Next”, it’s far less configurable – matter of fact, I couldn’t find any settings in Preference that control the Up Next functionality.

Up Next? More like Up Yours, Apple.

“Up Next”? More like “Up Yours”, right Apple?

Changing the keyboard shortcut for MiniPlayer (e.g. Command-Shift-M to Command-Option-M) – why would Apple do something stupid like this? Do you not understand and value muscle memory in consistent User Interface design? The MiniPlayer was one of iTunes’s killer features, and part of its utility was its hotkey which a MacWorld article even included in its 10 essential iTunes keyboard shortcuts last year.

iTunes 10 MiniPlayer

Also, why hide the currently playing track information in the MiniPlayer?! Upgrades should be improvements that add or at least refine features, not remove useful features that existed in previous versions! Now, in order to see what track is playing or how long it is or to seek around in the song, I have to click on the cover art icon in the MiniPlayer, first?

iTunes 11 MiniPlayer

I mean, look at this. What is this garbage?

I know Apple recently fired Scott Forstall. I do hope Apple fires whoever the Product Manager is in charge of this iTunes 11 release. It looks like they were more concerned with the redesign of the application icon and overall visual design of the application and totally ignored the ridiculously stupid changes that were being made to its functionality.

iTunes 10 to 11 icon WTF?

Wait, someone thought this was an improvement?

Shame on you, Apple. It’s a good thing Steve Jobs is dead, because this kind of embarrassing change would have KILLED him.

Charlie at WordCampNYC 2012

We arrived a little late, but we made it. Here’s Charlie at WordCampNYC 2012.

Charlie at WordCampNYC 2012

How young is too young … for WordCamp?

WordCamp NYC

I was thinking about attending WordCamp NYC 2012 this coming June, and I thought, “Hey, Charlie has a WordPress blog, maybe she’d like to attend, too.”

I mentioned this to Samantha, and it brought up the question: Is she too young to attend? I’m not so sure. Charlie’s quite bright for her age, she’s occasionally shown an interest in blogging, and seeing the wide range of possibility by meeting and interacting with other bloggers could give her new ideas, inspire her, generate additional interest, etc.

What say you, blogosphere? Are there likely to be too many age-inappropriate topics discussed within earshot? Will she just be ignored and written off as too young to interact with, and therefore be bored and discouraged?

I may not have the plumbing, but I have a brain

Especially in election years, the following clich

Oh, Starbucks, you really know how to get my attention!

Anyone who knows me should understand why this ad on the New York Times website home page caught my attention …

Starbucks blonde roast

(Zoom! Enhance!)

Starbucks Blonde Roast

I’m sure it tastes like garbage like all other Starbucks coffee, but it did get my attention.

Reminds me of that old clich

2011 In Review, Part 1

On the cusp between 2011 and 2012, I decided to take advantage of the much-maligned Facebook Timeline feature and briefly summarize some highlights from the past year.

January

Ted Williams, the man with the &dquot;golden radio voice&dquot;

January brought us the great story of Ted Williams, a homeless man with that deep radio-personality voice. It’s the kind of feel-good story that was a perfect way to start out the year. Or, at least try to …

Sadly, we learn of Bill Zeller‘s suicide, before which he posts an elaborate suicide note. I could write volumes on how I feel about this, but I’ll just leave well enough alone.

My friend Ian visited us in January, and his Prius wouldn’t start when he went to leave. I can now say that I’ve jump-started a Prius. With all those batteries, you’d think Toyota would have designed it to never need a jump start, right? Wrong.

I discovered that the Mobile Safari browser limits each open tab to 5 MB of memory.

In a fit of despair, I channeled my stress into a redesign of my blog.

Working briefly with Logan Zanelli, we launched Johnny B. Truant‘s new blog design.

Discovered an incredible YouTube video of Phish’s Meatstick from New Year’s 2010:

Egypt packed up its toys and went home, disconnecting the whole country from the Internet.

The North American blizzard dropped so much snow on us, it was the snowpocalypse.

I was able to get my email inbox down to 1,685 messages.

We tried to introduce the girls to Dungeons & Dragons. They both really seemed to like it.

February

Suzie turned 8, and Charlie turned 11. Both of them celebrate their birthday in February. This makes for a very hectic month, to say the least.

I was introduced to Van Canto’s version of Master of Puppets. A cappella heavy metal. Two great tastes that taste great together …

A brilliant bank robbery was committed, where the robber’s escape plan involved blending into a crowd of construction workers, crowdsourced from Craigslist, and the getaway vehicle was an inner-tube.

Through some friends-of-friends, I started doing some consulting work for BloomAvenue.com (which is now no longer). Later in the year, this would transform into me working from FromYouFlowers.com.

March

Good ol’ Charlie Sheen grabs a few minutes of fame with his “goddesses.”

Goodnight Dune

Julia Yu gave us Goodnight Dune, a fantastic parody of the age-old children’s classic of similar name.

The world reeled as Japan got pwnt by a tremendous earthquake and tsunami. The catastrophic event trashed the Fukushima nuclear power plants, and held the world’s attention with radiation scares.

Apple launched the iPad 2 tablet this month.

Mazda recalls a bunch of Mazda6’s because of some spiders nesting in its parts.

Blair River, the spokesman for the Heart Attack Grill, ironically dies from pneumonia, at the young age of 29.

Taylor Mali’s incredibly powerful poetry about the importance of teachers makes the rounds on the Internet, again:

The Passaic, Morris and Essex Counties were ravaged by intense flooding. Governor Chris Christie declared a state of emergency. Unfortunately, this won’t be the last time the area floods, this year.

Charlie and Suzie become “mages” at MagiQuest, starting with the one at the Funplex in East Hanover, NJ.

Facebook quietly starts working on Facebook Deals, most likely in response to Groupon’s tremendous success.

Facebook Deals

Lady Gaga reminds us that she was Born This Way. Matter of fact, we all are. Good to remind ourselves of this, once in a while.

My friend Maya returns to the United States, and I get to see her again after years since the last visit. It’s nice to catch up with old friends, even if it’s bittersweet.

Google finally launches their latest entry into the social web space with “Google +1.”

RSA, the company that’s well-known for creating one-time-password security fobs, gets hacked. This story will not end well …

April

Eric Whitacre leverages technology to create his “Virtual Choir 2.0.” This is the future I want to live in.

I published a HOWTO on tethering an Android phone via USB to MacOS X.

Epsilon has a data breach where millions of email addresses were acquired.

I did some DIY repair on my Mitsubishi Lancer, replacing the O2 sensor, so that the car would pass NJ state inspection.

We saw the US government nearly shut down because our politicians can’t agree on how to best overspend the American taxpayer dollar.

Steve Buscemi eyes

The eyes might be the windows to the soul, but Photoshopping Steve Buscemi’s eyes onto other people is a window into insanity. Thanks, Internet … everyone needs a new nightmare, every now and then.

My college alma mater, Rutgers, demonstrates that the “RU Screw” is still alive and well, by paying Snooki some $32k to perform.

In the “seriously, this happened?” department, the Med Peds Clinic of Fort Collins, Colorado, employed someone who thought it’d be a good idea to split a flu vaccine dose, by using half of the syringe and swapping out the needle. No, you can’t make this stuff up.

Just when I thought I lived in a sleepy little suburban town, tragedy strikes when someone brutally beats Frank the barber who dies from his injuries, right on Main Street in Bloomingdale.

Charlie's first communion

The four of us spent a week down in Atlantic City, at the Wyndham Skyline Tower.

Charlie has made her first communion at church this year, on Palm Sunday.

SETI stops operating the Allen Telescope Array. Millions of geeks cry out as they try to figure out what to do with their unused CPU cycles.

Adam Mansbach’s faux children’s book “Go the F— to Sleep” gets some press.

Sony’s PlayStation Network (PSN) gets hacked, officially. This won’t be the last time …

May

Just in case you forgot what the American “War on Terror” was all about, Osama bin Laden is finally dead. Little did Sohaib Athar realize at the time, but he live-tweeted the raid.

I finally broke down and bought myself a white iPad 2 (16 GB, Wi-Fi).

Nicktoons threatens to destroy more of my fond childhood memories, looting the corpse of the Voltron legacy.

The Library of Congress makes a wealth of old recordings available online.

Microsoft begins the process of acquiring Skype for $8.5 billion in cash. No shortage of “Sky.NET” jokes at this point.

Google launches its Music Beta service. Is it really the right time for a cloud-based music service?

We get to watch the FCC Commissioner Meredith Attwell Baker approve Comcast’s purchase of NBC, then go work for Comcast. No, I’m not kidding …

It’s finally official: I’m now working for FromYouFlowers.com as a Senior Solutions Architect.

His Holiness the Dalai Lama comes to Newark, New Jersey, and tries to tell Americans that we need to teach our kids ethics.

Watermelon farmers in China get screwed when they overdosed their crops with growth chemicals, resulting in exploding watermelons.

Sony’s PSN–you know, the Password Sharing Network–gets hacked again.

Macho Man vs Jesus

Harold Camping predicts the end of the world on May 21st, 2011. Sadly, the Macho Man Randy Savage dies on the 20th, just in time to stop Jesus from coming back to rapture us all. OHHHH, YEAHHHH!

Apparently some “faulty software” released 450 of the most dangerous inmates from California prisons. And people try and make fun of me for living in New Jersey … ??!

One of America’s largest defense contractors, Lockheed Martin, gets hacked. Allegedly, the breach may have involved the recently compromised RSA security technology.

The Ridgewood Cavaliers of Harmony, the barbershop society chapter I belong to, had a sing-out at Van Neste Square in Ridgewood, New Jersey, for Memorial Day.

June

Samuel L. Jackson, the quintessential Bad Mother F@#!ker, does the audiobook reading of “Go the F— to Sleep.” Naturally.

The girls have their Performing Arts Academy dance recital at the Shea Center on the William Paterson University campus.

Congressman Anthony Weiner resigns after a little sexting scandal, where he accidentally tweeted a link to a picture of his junk in his underwear. D’oh.

Neil Patrick Harris performs the opening number for the 2011 Tony Awards, pointing out that “Broadway’s not just for gays anymore.”

Web designers explore how repeating intervals that follow prime numbers in nature, specifically the cicada, can yield interesting visual effects, as well.

RIM’s stock drops in value by 20% overnight.

Michael Vick comes to Butler, New Jersey, to sign autographs, and winds up being greeted by protesters.

Google starts rolling out Google+.

Weird Al Yankovic releases his Lady Gaga parody, “Perform This Way.”

Britney Spears reminds us that she’s still a hottie, even after poppin’ out a baby, with her video for “I Wanna Go.” Hell, even if you don’t care for Britney, Guillermo Diaz who plays Scarface in Half Baked, appears in the video.

The state of Georgia enacts some anti-illegal-immigrant law, which totally screws the farmers in Georgia who can’t find enough affordable labor to harvest crops. If it wasn’t clear that illegal immigrants aren’t the problem before, it sure ought to be now.

Danous Estenor performs a heroic act of total badassery by lifting up a car, freeing a man who was trapped underneath.

Gary Foster made his alma mater Rutgers proud by embezzling some $19.2 million dollars. In America, you can only get away with it if you’re white …

***

Okay, it’s almost midnight … and I’m only up to July, so I’m going to post this now. Maybe I’ll get to do the second half of 2011, tomorrow.

Happy New Years, everyone!

MySQL replication connection error on non-standard ports

or, “Why SELinux was the worst thing to happen to Linux since LinuxThreads.”

Setting up MySQL master-slave replication is pretty straightforward. I’ve done it plenty of times. Similarly, running multiple MySQL server instances on the same host using mysqld_multi, really painless. Give each instance its own server-id and port to listen on, and that’s it.

In this case, the server that was being replicated was listening on port 3307. I could connect to it just fine using the mysql client using “-P 3307” as the replication user. It was clearly possible for the two machines to communicate over TCP port 3307. But, then, why was the MySQL slave I/O thread failing to connect to the master?

mysql> SHOW SLAVE STATUS \G
*************************** 1. row ***************************
               Slave_IO_State: Connecting to master
                  Master_Host: x.x.x.x
                  Master_User: repl
                  Master_Port: 3307
                Connect_Retry: 60
...
             Slave_IO_Running: Connecting
            Slave_SQL_Running: Yes
...
                Last_IO_Errno: 2003
                Last_IO_Error: error connecting to master 'repl@x.x.x.x:3307' - retry-time: 60  retries: 86400
...
1 row in set (0.00 sec)

Makes no sense, right? Right! Unless, of course, you’re on a machine–in this case, Red Hat Enterprise Linux 5.7–that has ever-so-helpfully screwed you by installing SELinux policies that restrict what ports mysqld can use, and has SELinux enabled by default.

How can you tell if SELinux is enabled and its policies are being enforced?

# grep 1 /selinux/enforce
1

If you get no output, then SELinux isn’t being enforced. But, if it is enabled, here’s how you can tell if mysqld‘s ability to make network connections is being restricted by a policy:

# semanage port -l | grep 3306
mysqld_port_t                  tcp      1186, 3306

You can tell if this is happening if you look in /var/log/audit/audit.log and see an entry like this:

type=AVC msg=audit(1320869122.773:56478): avc: denied { name_connect } for pid=2990 comm=”mysqld” dest=3307 scontext=user_u:system_r:mysqld_t:s0 tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket

(I bolded the elements to keep an eye out for.)

So, how do we remedy the situation? Personally, my first recommendation is to turn SELinux off. It is always more trouble than it’s worth. I’ve never heard of a single anecdote where it improved security or otherwise did anything useful.

Disable SELinux by editing /etc/selinux/config and change the line that reads “SELINUX=enforcing” to “SELINUX=disabled” and then reboot the system. There’s some caveats around disabling SELinux with respect to re-enabling it in the future, but you’re never going to do that if you have a choice, so who cares.

If you absolutely must leave SELinux enabled, because you’re actually burning in Hell and being tortured by some unreasonable and irrational organizational policy that requires it, then this is how you can add ports to the list of ports that mysqld will be allowed to use:

# semanage port -a -t mysqld_port_t -p tcp 3307

This command might take a few seconds to complete, and won’t generate any output if successful. You can check to see if it did what you wanted using the same command we used before, but this time the newly added port should be in the list, as well:

# semanage port -l | grep 3306
mysqld_port_t                  tcp      3307, 1186, 3306

To make sure this really did fix the problem, lets look at the MySQL replication status, again:

mysql> SHOW SLAVE STATUS \G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: x.x.x.x
                  Master_User: repl
                  Master_Port: 3307
                Connect_Retry: 60
...
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
...
                Last_IO_Errno: 0
                Last_IO_Error: 
...
1 row in set (0.00 sec)

And there you have it, replication configured on TCP port 3307, connected successfully, waiting for the master to send it updates, like it should be.

Hopefully, this has helped save you time, aggravation and much head-scratching as you tried to figure out why MySQL replication wasn’t working on a port other than 3306.

Got any other good gotcha stories like this? Share them in the comments below.

Business meals for self-employed, work-from-home consultants

I’d like to try and crowdsource an answer to a tax question I’ve had over the years but never felt like I had a good answer to:

IRS Topic 511, Business Travel Expenses, describes qualifying expenses while travelling. It is written with a very employee-centric tone, which isn’t particularly helpful for me as a self-employed consultant who works primarily out of his own home office.

1) Is New York City “close enough” to New Jersey to consider it the same “general area” or does travelling to New York City count as valid business travel in my case?

Suppose I have a client (one of many) for whom I travel, once a week, from my home office in New Jersey to their office in New York City. Since I earn the majority of my income working out of my home office in New Jersey, I think it’s appropriate to consider New Jersey my “tax home.” But, the tax topic is worded using the phrase “general area” which is vague. How is this supposed to be interpreted?

2) What does the IRS consider an “assignment”?

Topic 511 refers to temporary and indefinite “assignments” and talks about durations less and greater than one year. For the self-employed consultant, are clients considered “assignments”? Are individual projects for a client an “assignment”? I can’t find a definition for “assignment” in the IRS Tax Glossary.

Most clients of mine are of the “indefinite” kind, ones I intend to work with for at least a year or more. Individual projects for my clients generally span anywhere from less than a single day up to several months, but never a year or more.

I think this qualifies my one-day-a-week on-site visits as “short term travel” and therefore my travel expenses are deductible, but given sufficiently loose interpretations in the IRS’s favor, it makes me nervous. How is this supposed to be interpreted?

Naturally, I can arrive at my own conclusions and interpretations of what this all means. What I’d like to know is if anyone here has dealt with the IRS with respect to these two issues, and what their official stance is.

I suppose I could try and call the IRS and try to explain this to them and get an answer, but I’m not too keen on getting a verbal one-off interpretation by some call center representative. Getting this in writing is somewhat important, for obvious reasons.

November Rain? How about October Snow …

The northeast US got slapped with a nice blizzard this last week of October 2011, right on Halloween weekend. There are still green leaves (!) on the trees, and the weight of the snow has taken out many tree limbs and wires, resulting in Governor Christie declaring a statewide state of emergency. Millions of people across New England are without power, up to 600,000 of them in New Jersey alone.

Making the best out of the situation, our girls got bundled up and went outside to play in the snow. Here’s a short video I took of them: