memonic

16 Linux Server Monitoring Commands You Really Nee... - Input Output

Save

Want to know what's really going on with your server? Then you need to know these essential commands. Once you've mastered them, you'll be well on your way to being an expert Linux system administrator. Depending on the Linux distribution, you can

General Puppet errors

Save

Puppet is a wonderful system automation tool, however the learning curve can be a little steep. We've collected some of the errors messages and "strange" behaviour you may come across together with explanations to help overcome these hurdles and boost adoption of this fabulous tool.

If you have any useful errors and explanations, please do send them in and we'll update this article.

General Puppet errors

These errors happen on current or popular versions of Puppet.

Failed to retrieve current state of resource: Could not retrieve information from source(s)

err: //test/File[/tmp/foo]: Failed to retrieve current state of resource: Could not retrieve information from source(s) puppet:///test/foo at /etc/puppet/modules test/manifests/init.pp:5

The file cannot be found on the puppetmaster. Check that the source file exists on the puppetmaster and is readable by the puppetmaster process. Note that the file is actually required to be in /etc/puppet/modules/test/files/foo.

If the error says ... from source(s) test/foo then you have omitted puppet:/// from your manifest - check that it says something like:

source => "puppet:///test/foo"

Could not retrieve information from environment production source(s) puppet://

err: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve
information from environment production source(s) puppet://foo/plugins

You are using pluginsync (good) however there are no plugin directories on the puppet master - see this page. Easiest solution is to ensure that you have at least one module with a "lib" subdirectory.

Could not request certificate: undefined method `closed?'

err: Could not request certificate: undefined method `closed?' for nil:NilClass Exiting; failed to retrieve certificate and watiforcert is disabled

This one is normally very simple - you've forgotten to run puppetd as root and so it does not have permission to read the certificates.

This can also occur if there is a firewall preventing the puppet client and
puppetmaster from talking. (Thanks to Anand Kumria).

Change from absent to file failed

err: //test/File[/tmp/missing/foo]/ensure: change from absent to file failed: Could not set file on ensure: No such file or directory - /tmp/missing/foo at /etc/puppet/modules/test/manifests/init.pp:5

"No such file or directory? But you are supposed to make it!" is a typical response. The problem here is that the parent directory /tmp/missing does not exist. Either tell puppet to create the missing directory (note that you don't need an explicit dependency as puppet will do that for you) or resolve this in a different manner (for example by depending upon a package which creates the parent directory).

Change failed ... Could not find server

err: //test/File[/tmp/foo]/content: change from {md5}068008008418dff20750a94336318974 to {md5}8db2d67767577c70b1251fd80ad32943 failed: Could not find server puppet

The error here was that a filebucket had been set however the server name was set to "puppet" rather than the real name:

filebucket {
puppetmaster: server => "puppet1.example.com"
}

Could not retrieve catalog: can't convert nil into String

err: Could not retrieve catalog: can't convert nil into String at /etc/puppet/modules/test/manifests/init.pp:26 on node test.example.com

This error message happens when a template file is missing. As of v0.25.0, it prints a more helpful error message:

Could not find template 'test/foo' at /etc/puppet/modules/test/manifests/init.pp:26 on node test.example.com

undefined method `closed?' for nil:NilClass

err: Could not retrieve catalog from remote server: undefined method `closed?' for nil:NilClass

This error can occur when you have a bad error in a template - for example missing a closing quote. Double check any recent changes.

certificate verify failed

err: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
err: /File[/var/lib/puppet/lib]: Failed to retrieve current state of resource: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed Could not retrieve file metadata for puppet://puppet.example.com/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

The server certificate that puppet has and the one that the puppetmaster is using differ. On a pure puppet node one simple way is just to remove the current SSL information and start again:

find /var/lib/puppet -type f -print0 |xargs -0r rm

Also ensure that the client and server agree what the current time is (otherwise the certificate created may not be valid on the other machine).

no certificate found and waitforcert is disabled

warning: peer certificate won't be verified in this SSL session
Exiting; no certificate found and waitforcert is disabled

You just need to sign the certificate on the puppetmaster:

puppetca -l (returns a list of unsigned certificates)
puppetca -s node1.example.com

Could not retrieve catalog from remote server

err: Could not retrieve catalog from remote server: No such file or directory - /var/lib/puppet/client_yaml/catalog

This can occur if /var/lib/puppet/client_yaml/ is missing on the client (so that the catalog file cannot be written).

Could not render to pson: invalid utf8 byte

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not render to pson: invalid utf8 byte: '�'

One of your templates contains a strange character. You can use "od -c filename" and look for the odd characters.

Could not retrieve catalog from remote server: wrong header line format

err: Could not retrieve catalog from remote server: wrong header line format

One of your templates contains invalid syntax. Look for unintentional or invalid "<%" and ">%" strings. If you need to put these strings in templates you can use "<%%" and ">%%".

Run of Puppet configuration client already in progress

notice: Run of Puppet configuration client already in progress; skipping

Puppet believes that it is already running. If this isn't the case then you may have a stale lock file. Check with "ps axf" to see if puppetd is running, if it isn't then delete the lockfile (/var/lib/puppet/state/puppetdlock is likely location).

You cannot specify more than one of content, source, target

err: Could not run Puppet configuration client: You cannot specify more than one of content, source, target at /etc/puppet/modules/examples/manifests/init.pp:8

This is self-explanatory, what might not be obvious is how to fix this (for example if you are overriding a File resource previously defined by a source attribute which you now which to use a content with. Simply set "source => undef".

Cannot override local resource on node

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Exported resource Opsviewmonitored[foo] cannot override local resource on node bar.example.com

You have a duplicate definition. This maybe due to an old node which you need to run puppetcleannode on. Search the database like this: "select hosts.name from hosts,resources where restype='Opsviewmonitored' and title='foo' and hosts.id = resources.host_id;"

Version specific error messages

These errors happen on more specific versions of Puppet.

Could not intern from pson: Could not convert from pson: Could not find relationship target ''

err: Could not retrieve catalog from remote server: Could not intern from pson: Could not convert from pson: Could not find relationship target ''

This is a bug specific to v0.25.1.

Error 400 on SERVER: No support for http method POST

err: Could not retrieve catalog from remote server: Error 400 on SERVER: No support for http method POST

Ensure that you are not running a v2.7 client against a v2.6 master. We suggest downgrading the client in this case to v2.6 as v2.7 can introduce changes in behaviour that are best dealt with as part of a planned upgrade.

InstantWatcher - Whats on Netflix

Save

The Netflix Arrival List sucks, its too personalized. Use InstantWatcher to find out whats new and hot.

RAID0 ephemeral storage on AWS EC2 - Gabriel Weinberg's Blog

Save

If you're thinking of doing RAID0 (disk striping) on the ephemeral storage disks attached to an EC2 instance, this post is for you. First I'll go through some directions since I didn't find great ones elsewhere when trying to do it myself. Then I'll get to why I wanted to do it in the first place.

Amazon EC2 Instances - Supported Filter Attributes

Save

filter

 
Filter Name Description

architecture

Instance architecture.

Type: String

Valid Values: i386 | x86_64

availability-zone

Instance's Availability Zone.

Type: String

block-device-mapping.attach-time

Attach time for an Amazon EBS volume mapped to the instance, e.g., 2010-09-15T17:15:20.000Z

Type: xsd:dateTime

block-device-mapping.delete-on-termination

Whether the Amazon EBS volume is deleted on instance termination.

Type: Boolean

block-device-mapping.device-name

Device name (e.g., /dev/sdh) for an Amazon EBS volume mapped to the instance.

Type: String

block-device-mapping.status

Status for an Amazon EBS volume mapped to the instance.

Type: String

Valid Values: attaching | attached | detaching | detached

block-device-mapping.volume-id

ID for an Amazon EBS volume mapped to the instance.

Type: String

client-token

Idempotency token you provided when you launched the instance.

Type: String

dns-name

Public DNS name of the instance.

Type: String

group-id

ID of a EC2 security group the instance is in. This filter does not work for VPC security groups (instead, use instance.group-id).

Type: String

group-name

Name of a EC2 security group the instance is in. This filter does not work for VPC security groups (instead, use instance.group-name).

Type: String

image-id

ID of the image used to launch the instance.

Type: String

instance-id

ID of the instance.

Type: String

instance-lifecycle

Whether this is a Spot Instance.

Type: String

Valid Values: spot

instance-state-code

Code identifying the instance's state. A 16-bit unsigned integer. The high byte is an opaque internal value and should be ignored. The low byte is set based on the state represented

Type: Integer

Valid Values: 0 (pending) | 16 (running) | 32 (shutting-down) | 48 (terminated) | 64 (stopping) | 80 (stopped)

instance-state-name

Instance's state.

Type: String

Valid Values: pending | running | shutting-down | terminated | stopping | stopped

instance-type

Type of instance (e.g., m1.small).

Type: String

instance.group-id

ID of a VPC security group the instance is in. This filter does not work for EC2 security groups (instead, use group-id).

Type: String

instance.group-name

Name of a VPC security group the instance is in. This filter does not work for EC2 security groups (instead, use group-name).

Type: String

ip-address

Public IP address of the instance.

Type: String

kernel-id

Kernel ID.

Type: String

key-name

Name of the key pair used when the instance was launched.

Type: String

launch-index

When launching multiple instances at once, this is the index for the instance in the launch group (e.g., 0, 1, 2, etc.).

Type: String

launch-time

Time instance was launched, e.g., 2010-08-07T11:54:42.000Z.

Type: xsd:dateTime

monitoring-state

Whether monitoring is enabled for the instance.

Type: String

Valid Values: disabled | enabled

owner-id

AWS account ID of the instance owner.

Type: String

placement-group-name

Name of the placement group the instance is in.

Type: String

platform

Use windows if you have Windows based instances; otherwise, leave blank.

Type: String

Valid Value: windows

private-dns-name

Private DNS name of the instance.

Type: String

private-ip-address

Private IP address of the instance.

Type: String

product-code

Product code associated with the AMI used to launch the instance.

Type: String

ramdisk-id

RAM disk ID.

Type: String

reason

Reason for the instance's current state (e.g., shows "User Initiated [date]" when you stop or terminate the instance). Similar to the state-reason-code filter.

Type: String

requester-id

ID of the entity that launched the instance on your behalf (e.g., AWS Management Console, Auto Scaling, etc.)

Type: String

reservation-id

ID of the instance's reservation. A reservation ID is created any time you launch an instance. A reservation ID has a one-to-one relationship with an instance launch request, but can be associated with more than one instance if you launch multiple instances using the same launch request. For example, if you launch one instance, you’ll get one reservation ID. If you launch ten instances using the same launch request, you’ll also get one reservation ID.

Type: String

root-device-name

Root device name of the instance (e.g., /dev/sda1).

Type: String

root-device-type

Root device type the instance uses.

Type: String

Valid Values: ebs | instance-store

source-dest-check

Whether the instance performs source/destination checking. A value of true means checking is enabled, and false means checking is disabled. The value must be false for the instance to perform Network Address Translation (NAT) in your VPC.

Type: Boolean

spot-instance-request-id

ID of the Spot Instance request.

Type: String

state-reason-code

Reason code for the state change.

Type: String

state-reason-message

Message for the state change.

Type: String

subnet-id

ID of the subnet the instance is in (if using Amazon Virtual Private Cloud).

Type: String

tag-key

Key of a tag assigned to the resource. This filter is independent of the tag-value filter. For example, if you use both the filter tag-key=Purpose and the filter tag-value=X, you get any resources assigned both the tag key Purpose (regardless of what the tag's value is), and the tag value X (regardless of what the tag's key is). If you want to list only resources where Purpose=X, see the tag:key filter later in this table.

For more information about tags, go to Using Tags in the Amazon Elastic Compute Cloud User Guide.

Type: String

tag-value

Value of a tag assigned to the resource. This filter is independent of the tag-key filter.

Type: String

tag:key

Filters the results based on a specific tag/value combination.

Example: To list just the resources assigned tag Purpose=X, then specify:

--filter tag:Purpose=X

Example: To list just resources assigned tag Purpose=X OR Purpose=Y, then specify:

--filter tag:Purpose=X --filter tag:Purpose=Y

virtualization-type

Virtualization type of the instance.

Type: String

Valid Values: paravirtual | hvm

vpc-id

ID of the VPC the instance is in (if using Amazon Virtual Private Cloud).

Type: String

hypervisor

Hypervisor type of the instance.

Type: String

Valid Values: ovm | xen

 

Puppet at Loggly

Save

Jordan Sissel. hacker. Loggly, Inc. A source of Truth feeds the Model and results in an applied configuration. I never use ‘import'. I always let puppet determine the path of a class. Your puppet repository should look like this: Puppet lets you abstract things away from the madness of your OS, let it.

Comments (1)

Toni Birrer

Toni Birrer Jan 22, 2012

An excellent list of best practices for puppet.

Setting up a Storm cluster - GitHub

Save

This page outlines the steps for getting a Storm cluster up and running. If you're on AWS, you should check out the storm-deploy project. storm-deploy completely automates the provisioning, configuration, and installation of Storm clusters on EC2. It also sets up Ganglia for you so you can monitor CPU, disk, and network usage.

ZooKeeper Administrator's Guide

Save

This section contains information about deploying Zookeeper and covers these topics: The first two sections assume you are interested in installing ZooKeeper in a production environment such as a datacenter. The final section covers situations in which you are setting up ZooKeeper on a limited basis - for evaluation, testing, or development - but not in a production environment.

Apache ZooKeeper - Overview

Save

ZooKeeper allows distributed processes to coordinate with each other through a shared hierarchical name space of data registers (we call these registers znodes), much like a file system. Unlike normal file systems ZooKeeper provides its clients with high throughput, low latency, highly available, strictly ordered access to the znodes.

Writing a Confluence Plugin for a SaaS Product | Atlassian Blogs

Save

This is a guest post by Brian Pugh of LucidChart, an online diagramming solution that makes diagramming fast and easy. In this post, Brian talks about the process of writing a Confluence plugin for LucidChart.

 

As you probably know, Confluence has a flexible plugin model that allows third parties to add new functionality and features. Typically, creating a Confluence plugin has involved adding new features in the plugin itself. However, as more companies are providing their software as a service (SaaS), it doesn’t always make sense to re-implement functionality in a plugin that already exists in a SaaS product. Instead, a plugin can be written to integrate the existing SaaS product with Confluence.

Why SaaS?

Integrating Confluence with an existing SaaS offering can provide a great experience for users.  If you already have a SaaS product, you probably also have a substantial user base.  Those users will want to access the data from their existing account within Confluence rather than have to manage one account for Confluence and another account for use outside of Confluence (either using your product stand-alone or with another service such as JIRA).

Furthermore, integrating Confluence with your existing SaaS product gives you flexibility to grow your SaaS offering. As you improve the SaaS offering, those improvements will be available immediately to your Confluence users without you needing to re-write the plugin and duplicate code and without existing users having to update their plugin.

This post describes the approach we took at LucidChart to provide our users a great Confluence experience with our diagramming SaaS offering.

What is LucidChart?

First, a little about LucidChart. LucidChart is a collaborative diagramming application that makes it easy to draw flowcharts, UML diagrams, UI mockups, network diagrams, org charts and more. With LucidChart’s real-time collaboration, you can work together with an unlimited number of others to create and edit diagrams, with changes merged and synced instantaneously — great for team collaboration.

The LucidChart plugin for Confluence allows you to create new diagrams, insert existing diagrams from your LucidChart account, or even import Microsoft Visio documents into your wiki. The plugin described in this post is available on the Atlassian Plugin Exchange.

Authorization with OAuth

When creating a Confluence plugin for a SaaS product, one of the first challenges encountered is determining how Confluence will gain access to the user’s data in the SaaS product.  For the LucidChart plugin, we turned to OAuth to solve this problem. OAuth is an industry standard for delegated authorization. A very brief description of an OAuth 3-legged authorization flow is given below.  The details of the protocol are beyond the scope of this post, but can be found at oauth.net.

OAuth allows a consumer (in our case, the Confluence plugin) to request authorization from a service provider (LucidChart in our case) by redirecting the user to a page provided by the service provider where the user is asked whether she would like to allow the consumer to access her data.  If the user grants access, she will be redirected back to the consumer and the consumer will be given a token that allows it to access data from the service provider.

Adding the Menu Item

With a basic understanding of the OAuth authorization flow, let’s dive in to how we created the plugin. We want users in Confluence to be able to quickly and easily insert a LucidChart Diagram on a page. To do that, we’ll insert a menu item in the Confluence “Add” drop down menu labeled “LucidChart Diagram”. Menu items are added by creating a Web UI module, which is done by adding an entry to atlassian-plugins.xml.

Click to get code

Each section of the web-item definition is described in detail in the Confluence Web UI module documentation. With this web-item definition, a menu item will appear in the “Add” drop down for spaces and pages with the text “LucidChart Diagram” and the icon “lucid.png”. When the item is clicked, it will send the user to the relative URL below with $page.id replaced by the id of the Confluence page the user was on when he clicked the menu item:

/plugins/lucidchart/insertDiagram.action?pageId=$page.id

Creating the XWork Module to Handle Authorization

Now we need to define what happens when the user arrives at the URL we provided in the Web UI module. An  XWork module will do the trick.  We define the XWork module in the atlassian-plugins.xml.

Click to get code

This snippet defines a WebWork action “insertDiagram” for the URL we provided to the web-item.  The Java class InsertDiagram.java extends ConfluenceWebActionSupport and overrides the execute() method which will be called when the user arrives at the URL for this action.

When a user clicks on our web-item and arrives at our WebWork action, we need to first determine whether the user has already authorized Confluence to access her LucidChart account and data (i.e. determine if an OAuth Access Token is already available). If the user has not yet authorized Confluence to access her LucidChart data, we initiate the three-legged OAuth flow. This means that the user will be redirected to a page hosted at http://www.lucidchart.com where she can grant or deny Confluence access to her data.

Grant Confluence Access to Your LucidChart Data

If the user grants access, she will be redirected to a callback URL provided previously as part of the OAuth flow.  In our case, the callback will be the URL of the WebWork action we defined previously.

When the execute method is called this time, rather than redirecting to the LucidChart authorization page, we will complete the OAuth flow to obtain and store an access token.

At this point, we have an OAuth access token that can be used to sign requests to LucidChart thus allowing us to make API calls from our plugin to LucidChart to access the user’s LucidChart data. Future requests made by the user to add a LucidChart diagram will not need to go through this authorization process because the stored access token can be used until it is explicitly revoked by the user.

Displaying the User’s LucidChart Document Manager

Now that the user has authorized Confluence to access his LucidChart data, we display a screen that allows the user to make a new diagram or choose an existing diagram.

In our atlassian-plugins.xml XWork module definition, we included a result for “success”.  When the execute() method of the InsertDiagram WebWork action completes the OAuth flow, it will return the string “success” which will cause the success result to be executed.  In our case, we defined that as a velocity template called document-list.vm.  The velocity template will render the Confluence header and footer, then include in an iframe the user’s “document list” as served from LucidChart servers.  The velocity template is:

Click to get code

The page allows the user to select an existing diagram from his account or create a new diagram.

Creating a new LucidChart Diagram in Confluence

The OAuth access token was used to make a signed oauth request in the iframe.   Because we provided a signed OAuth request, the LucidChart servers render a view of the LucidChart document list for the specific user making the request.  In the URL for the document list, we also included a callback for the LucidChart servers to use when a user selects a diagram to add to Confluence or creates a new diagram.  If we “view source” on the page, we would see that the URL for the iframe looks something like:

https://www.lucidchart.com/documents/external?oauth_signature=vOy8rrxOq5mZu1PJS%2FwxLLeb33g%3D&oauth_version=1.0&oauth_nonce=1517750262&oauth_signature_method=HMAC-SHA1&oauth_consumer_key=218c0c6ff562b39cb2f4d20bd90ff34f&oauth_token=f9bb655ea3906ab009c4e97c195877a82c68cde6&callback=http%3A%2F%2Fconfluence%2Fplugins%2Flucidchart%2FattachDiagram.action%3FpageId%3D557064&oauth_timestamp=1318615334

Notice that the URL has the OAuth parameters as well as a callback parameter which when URL decoded is:

http://confluence/plugins/lucidchart/attachDiagram.action?pageId=557064

When a users clicks to insert a document or creates a new document, the user will be redirected to the callback with a documentId parameter added to the URL.  For example, if the user clicks “Insert document” for document id 12345, he would be redirected to:

http://confluence/plugins/lucidchart/attachDiagram.action?pageId=557064&documentId=12345

Attaching the Diagram to the Page

When the user has selected a diagram to attach, we saw that the user is redirected to a callback that was provided in the URL for the iframe.  We now need to define a second XWork module for that callback.  The “attachDiagram” action is defined in the atlassian-plugins.xml as follows:

Click to get code

The execute() method of DiagramAttacher.java will pull the documentId from the URL and use it to request an image of the document from the LucidChart servers (again, using an OAuth signed request to obtain authorization to the image).  The DiagramAttacher will then attach each page of the diagram to the appropriate Confluence page with a unique name:

Click to get code

The diagram will now exist as an image attached to the Confluence page

Display the Diagram in the Page

While having the diagram attached to the page is okay, we’d like to provide a nice viewer for the diagram that allows zooming, panning and moving through the pages of a diagram.  To do this, we need to create a Confluence Macro Module.  When a user chooses to add a LucidChart diagram to a page, we will not only attach the images to the page as previously described, but also include wiki markup for a custom macro to view the diagram:

{lucidchart:name=testDiagram-181-74385522|id=4e443bc3-e7d8-4ea7-83d2-1a840ad07a84|pages=1|width=700|height=700}

The Macro Module will parse this markup and insert the appropriate diagram in a viewer.  As we did with the Web UI and XWork modules, we define the Macro module in the atlassian-plugins.xml:

<macro name="lucidchart" key="lucidchart"
class="com.lucidchart.confluence.plugins.LucidMacro" key="lucidchart">
</macro>

The Java class LucidMacro.java  extends com.atlassian.renderer.v2.macro.BaseMacro and overrides the execute() method.  The execute() method’s signature is:

public String execute(Map params, String body, RenderContext renderContext)

We can use params to get the parameters of our macro (name, id, pages, width, height).  Using those parameters, we can get a list of the URLs for the attached images that should be displayed:

Click to get the code

The list of attachment URLs to display is put in the velocity context so it can be used in a velocity template to render the diagram viewer:

Map<String, Object> context = MacroUtils.defaultVelocityContext();
context.put("attachments", attachments);
return VelocityUtils.getRenderedTemplate("templates/lucidchart/viewer.vm", context);

Viewer.vm can access the attachments and send them to a JavaScript diagram viewer:

Click to get the code

The result is that the diagram selected by the user is displayed in the Confluence page with the ability to zoom, pan and move from page to page.

LucidChart Viewer in Confluence

Conclusion

Creating a Confluence plugin for a SaaS offering is a simple and effective way to add value for your users.  With the flexibility provided by the Confluence plugin model and industry standards such as OAuth, it is straightforward for any SaaS offering to give their users the ability to quickly use their data in Confluence.

Note:  We were able to use the same approach to create a JIRA plugin for LucidChart

(1 - 10 of 563)