How long does it take to re-build indexes on your Author?

It just took 1 whole week to re-build the indexes on our live Author!!! Incredible.

Midnight at the lost and found

Have you worked out that TAR Optimiser does not run at midnight?

Normally, the OOTB default for running the TAR optimiser is 2am - 5am. This can be changed in the repository.xml/workspace.xml files. But, if you specify a start time of 00:00 it won't run.

I'm sure I've posted this elsewhere but just to re-iterate you can make the TAR optimiser run faster & do more work by reducing the optimizeSleep parameter. We've managed to get away with 0.25 without any noticeable performance impact to the live servers (CQ 5.3).

Logging Activates, Deactivates and Deletes

To keep a log of who has activated or deactivated a page, add this to your logging :-

Log Level: Debug

Logger: com.day.cq.replication.impl.queue.ReplicationJob

You will then get entries like this in your log file :-

01.05.2012 13:58:07.460 *DEBUG* [123.12.34.56 [0005877072284] POST /bin/replicate.json HTTP/1.1] com.day.cq.replication.impl.queue.ReplicationJob Creating replication job null (action: ReplicationAction{type=ACTIVATE, path[0]='/content/mysite/en/blah/blah', time=1335877072325, userId='martin', revision='null'}) with content ReplicationContentFacade{path='/var/replication/data/g4e8029a-deec-87c7-8176-6d4eadd8fda5/84/84a8bf03-a58a-482d-8186-a7d2819ca365', contentType='application/cq5-replication-durbo', contentLength=9492}

Handling DELETEs which flush the dispatcher cache

I have been working on the following problemette, which was posted to the DAY-CQ mailing list on Google.com :-

Hi CQ Community,

Does anyone know how to stop the dispatcher invalidating on a DELETE command down a path?

The reason why I ask is because we have a lot of usergenerated content which is being reverse replicated. When the UGC is moved, for security, from /content/usergenerated to /content/mysite/blah/blah, then the /content/usergenerated/... node is deleted on the publish server. Each of these delete commands triggers the flush agent.

I have tried defining a rep:policy to deny jcr:all on a user in /content/usergenerated/. This works for node additions but, deletions are not recognised. So I cannot stop it here.

I have tried to alter the configuration in the /invalidate section of dispatcher.any file to no avail. Is this sdection defining what objects get invalidated rather than what objects trigger an invalidation?

I also noticed that in the release notes of the dispatcher the following, which makes me think that invalidate on delete might be hard-wired ...

Issues resolved in 4.0.5:
25169 - Support flush on every write

Any help would be greatly appreciated!

This turned out to be a product bug - so it's in the queue to fix for a future version. Great!

In the meantime, we have come up with a workaround which is to implement a HTTP rewrite rule based upon the CQ-Handle /content/usergenerated (we don't need to flush on any changes in this area).

The key to intercepting the /dispatcher/invalidate.cache was to catch the CQ-Handle header /content/usergenerated and then to set CQ-Action to "TEST". This nullifies the request.

WebDAV client

Try this one : http://www.bitkinex.com/webdavclient

Reverse Replication woes - solved

Hot off the press/keyboard (i.e. not fully tested). With the help of an Adobe support engineer in Basel and an on-site Adobe consultant we discovered what the root cause of the reverse replication problem was.

Namely, that when a user voted in a poll, the new vote AND ALL previous votes were being reverse replicated. This caused a MASSIVE workload on the Author because each node in the /var/replication/outbox did not contain 1 corresponding vote; it actually contained ALL of the votes including the new additional one. This explains why the Author would take 20 minutes to process just 10 nodes in the outbox.

The root cause was in the structure we were using (has been abbreviated):-

/content
   /usergenerated
      /somepoll
         /poll1 [cq:Page]
            /jcr:content [cq:PageContent]
               /question
                  /answers
                     /1
                        /12423434
                        /12312323
                     /2
                        /23463456

Each vote is added under the /answers node as type "nt:unstructured" with the various properties. But, on each submission of a vote, the custom code (a custom SlingPostServlet) was setting the 3 magic properties (cq:distribute, jcr:lastModified & jcr:lastModifiedBy) on "/poll1/jcr:content". This causes the page "poll1" to be marked for reverse replication - and with it, all it's sub-nodes (aggregated).

The solution was to change the nodes that get created to individual pages themselves as follows :-

/content
   /usergenerated
      /somepoll
         /poll1
            /question
               /answers
                  /1
                     /12423434 [cq:Page]
                        /jcr:content [cq:PageContent]
                     /12312323 [cq:Page]
                        /jcr:content [cq:PageContent]
                  /2
                     /23463456 [cq:Page]
                        /jcr:content [cq:PageContent]

And, then to ensure that the 3 magic properties are created on the jcr:content node of each vote node. Note: DO NOT have a jcr:content node anywhere in the intermediate hierarchy because this interferes with the firing of the outbox manager (I think it sees a jcr:content node and assumes that there must be a page there but, because there isn't a page there, then it aborts - and nothing appears in the outbox. I suffered with this problem when I kept "poll1/jcr:content" in the path (i.e. /content/usergenerated/somepoll/poll1/jcr:content/question/answers/1/12423434/jcr:content).

NB, due to our environment, we needed to use a custom SlingPostServlet and start the reverse replication in our project. However, the above structure should work with the normal OOTB page manager activated reverse replication.

The HTML form to post these votes would be something like this :-

<form action="/content/usergenerated/somepoll/poll1/question/answers/1/123456789" method="post" enctype="multipart/form-data">
<input type="hidden" name="./jcr:primaryType" value="cq:Page" />
<input type="hidden" name="././jcr:content/jcr:primaryType" value="cq:PageContent" />
<input type="hidden" name="././jcr:content/answer" value="my_chosen_answer" />

<input type="hidden" name=":redirect" value='/content/website/thankyou.html' />
<button type="submit">Submit</button>
</form>

Reverse Replication woes

So, in my previous post I said how wonderful FP37434 is (the replication stabilisation FP). Unfortunately, it did not solve our problem and we now have a large volume of content to reverse replicate (~50k nodes in /var/replication/outbox across all our publish servers).

We are currently facing 2 problems. When the RR agent polls, the publish server with FP37434 exhibits a huge native memory leak (approx 8GB of native memory is being claimed) causing a great deal of paging on the system.

When we batch this down to only 10 items in the outbox, we noticed that the author takes 30 minutes to process 10 nodes.

Adding extra logging (com.day.cq.replication.content.durbo) at DEBUG level shows that the Author is doing valid work for 30 minutes processing just 10 nodes from the outbox.

It turns out that when a node is added to /content/usergenerated/path/to/something then CQ appears to be adding all of the pre-existing sibling nodes in the newly created node under /var/replication/outbox. You can see this by analysing nodes inside the outbox. This is why 10 nodes takes 30 minutes for the author to process - because it's actually unpacking 10000 nodes.

This probably also explains why our CQ author is performing slowly.

Hopefully, I will remember to post the solution here when we get to it ... :-)