tag:blogger.com,1999:blog-68987844549569029472024-03-05T10:16:25.239+00:00Hub of TechAdel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.comBlogger37125tag:blogger.com,1999:blog-6898784454956902947.post-68097410267740146742023-08-14T22:35:00.006+00:002023-12-22T18:26:53.030+00:00Importing Trusted Certificate to all your JVM cacertsSometimes when you work in a corporate company there are proxies that require certificates to validate Java applications.
Usually your company will provide a valid certificate that will validate any requests coming in and going out from your local machine.
In order to use your Java applications successfully with your corporation's certificate you need to import it using keytool to the cacerts file for your installed JVM. <div><br /></div><div>The problem is however you may have multiple JVMs installed and the location may differ from machine to machine.
The following bash script locates all the cacerts on your machine and adds the appropriate certificate to them.
<pre>#!/usr/bin/env sh
PROXY_CERT="${HOME}/Your_Company_Proxy_CA.cer"
KEYSTORE_ALIAS="proxy-root"
echo "Finding cacerts..."
KEYSTORES=$(find / -name cacerts -type f -print 2>/dev/null)
while IFS= read -r KEYSTORE
do
echo "Finding alias ${KEYSTORE_ALIAS} from JDK Keystore ${KEYSTORE}"
sudo keytool -list -alias ${KEYSTORE_ALIAS} -keystore "${KEYSTORE}" -storepass changeit -v && {
echo "Deleting alias ${KEYSTORE_ALIAS} from JDK Keystore ${KEYSTORE}"
sudo keytool -delete -alias ${KEYSTORE_ALIAS} -storepass changeit -noprompt -keystore "${KEYSTORE}"
} || echo "Adding cert to JDK Keystore ${KEYSTORE}"
sudo keytool -import -trustcacerts -storepass changeit -noprompt -alias ${KEYSTORE_ALIAS} -keystore "${KEYSTORE}" -file "${PROXY_CERT}"
done <<< "${KEYSTORES}"
</pre>
</div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0tag:blogger.com,1999:blog-6898784454956902947.post-69040134655147252952023-06-24T22:32:00.006+00:002023-06-24T22:39:15.307+00:00Presenting about Quarkus at the Java Meet Up<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWhWYBf7B1N7XmLMfmhKGQ7rd9bPe30UZjPPMUNWM8s2jS4cdY4WLNtNOqGXWP37ZkpNErosboI8GeFddgkRkkQ2ftNUhugDB_T29N7zG0VdXO7r6QcdWPnl1VFukzgpRgf5VcWqJDU0z2TNc4Q7btpK8MQbe_ZTXdFHoBvGjztYMa2o3CzAcPQTo5uHa9/s1600/dsc09395.jpg" style="display: block; padding: 1em 0px; text-align: center;"><img alt="" border="0" width="100%" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWhWYBf7B1N7XmLMfmhKGQ7rd9bPe30UZjPPMUNWM8s2jS4cdY4WLNtNOqGXWP37ZkpNErosboI8GeFddgkRkkQ2ftNUhugDB_T29N7zG0VdXO7r6QcdWPnl1VFukzgpRgf5VcWqJDU0z2TNc4Q7btpK8MQbe_ZTXdFHoBvGjztYMa2o3CzAcPQTo5uHa9/s1600/dsc09395.jpg" /></a></div>
Quarkus is a framework designed for Kubernetes to improve the startup time and memory footprint of Java apps running in docker containers
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0tag:blogger.com,1999:blog-6898784454956902947.post-24577483290292892272023-06-12T13:43:00.004+00:002023-06-12T13:45:03.069+00:00Recursive CTE to List Years Without GapsIf I have a requirement to list the years from a start year incrementing to the current year without any gap years, I can write the following simple SELECT statement within a CTE.
<pre>WITH FINANCIAL_YEARS AS (
SELECT 2020 AS FINANCIAL_YEAR
UNION SELECT 2021
UNION SELECT 2022
UNION SELECT 2023
)
</pre>
However its pretty easy to see how tedious this can be especially when trying to maintain it when a new year ticks over.
I found this neat trick using recursive CTEs to avoid gaps in dates and tweaked it to deal with year increments:<br />
<pre>WITH FINANCIAL_YEARS AS (
SELECT '2020-01-01'::DATE AS YEAR_DATE
UNION ALL
SELECT DATEADD(YEAR, 1, YEAR_DATE) AS YEAR_DATE
FROM FINANCIAL_YEARS
WHERE YEAR_DATE < DATEADD(YEAR, -1, DATEADD(MONTH, 6, CURRENT_DATE())
)
SELECT DATE_PART(YEAR, YEAR_DATE) AS FINANCIAL_YEAR FROM FINANCIAL_YEARS
</pre>
Its even simpler if you only want the year component or any other kind of numeric increment from a starting number:
<pre>WITH FINANCIAL_YEARS AS (
SELECT 2020 AS FINANCIAL_YEAR
UNION ALL
SELECT FINANCIAL_YEAR + 1 AS FINANCIAL_YEAR
FROM FINANCIAL_YEARS
WHERE FINANCIAL_YEAR < YEAR(DATEADD(MONTH, 6, CURRENT_DATE()))
)
SELECT FINANCIAL_YEAR FROM FINANCIAL_YEARS
</pre>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0tag:blogger.com,1999:blog-6898784454956902947.post-49940000222813052602022-12-01T00:58:00.008+00:002022-12-01T01:00:58.166+00:00Reporting on your After Hours using Git commit timestampsSometimes its hard to keep yourself accountable to the extra long hours you work, and most of the time you don't want to show off how late you've been working, but every now and then your manager would probably ask if you've been putting in the necessary hours.
<br/><br/>
If thats the case, git commits can give you a certain level of transparency provided you commit regularly.
<br/><br/>
The following bash script assumes you have all your git repositories cloned to a folder specified in GITFOLDER which should be updated to your local setup. It will run a simple "git log --author" command for a given USERNAME and list only the timestamps after the hours of 5pm to the hours of 6am:
<pre>
#!/usr/bin/env sh
USERNAME="<username>"
GITFOLDER="${HOME}/dev"
cd ${GITFOLDER}
FOLDERS=$(ls)
while IFS= read -r FOLDER
do
cd "${GITFOLDER}/${FOLDER}"
TIME_LOGS=$(git log --author="${USERNAME}" | grep "Date:" | grep -E '\s1[7-9]+\:|\s2[0-3]+\:|\s0[0-6]+\:')
while IFS= read -r TIME_LOG
do
if [ ! -z "$TIME_LOG" ]
then
COMMIT_DATE=$(echo ${TIME_LOG} | sed -e "s/Date: //g")
echo "${COMMIT_DATE} ${FOLDER}"
fi
done <<< "${TIME_LOGS}"
done <<< "${FOLDERS}"
</pre>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0tag:blogger.com,1999:blog-6898784454956902947.post-22740511928361501532022-11-25T03:22:00.010+00:002022-11-25T03:27:06.857+00:00Reporting on your After Hours in SnowFlakeMy current work at <a href="https://www.tyro.com/" target="_blank">Tyro</a> is predominantly in the Snowflake Data Warehouse creating reports, building queries, and maintaining the DDL.
<br /><br />
As such it is necessary to keep monitoring my constant usage as the team starts to grow and engineers become more accountable.
<br /><br />
One good technique is to query the built-in view:<pre>SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY</pre>
It consists of the USER_NAME, ROLE_NAME, and even the QUERY_TEXT of query that was run in snowflake.
<br /><br />
What we need in order to find out the number of after hours worked on a day to day basis is the START_TIME of the query that was executed. With that we can then filter by any queries run after 5PM (i.e. the 17th hour of the day), and the difference between max and min dates for each day to find out how many hours were worked after that time.
<pre>SELECT
USER_NAME,
ROUND(TIMESTAMPDIFF('MINUTE', MIN(START_TIME), MAX(START_TIME)) / 60, 2) AFTER_HOURS,
DAYNAME(MIN(START_TIME)) DAY,
MIN(START_TIME) START_DATE,
MAX(START_TIME) END_DATE
FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY
WHERE USER_NAME = '<user_name>'
AND DATE_PART('HOUR', START_TIME) > 17
GROUP BY
USER_NAME,
DATE_PART('DAY', START_TIME),
DATE_PART('MONTH', START_TIME),
DATE_PART('YEAR', START_TIME)
ORDER BY MIN(START_TIME) DESC;
</pre>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0tag:blogger.com,1999:blog-6898784454956902947.post-78828403829865459482021-12-10T07:06:00.821+00:002022-02-03T23:06:57.364+00:00Idempotency as a Process<div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEiGXcPDpLNk3o9NttWov4cTES-15DI78mzJ1dHJtM0aXkrCFx4PdDvpCExaZDvqwSr17K_Dgj5-P-71iOeSNHHK7BQ7YQ75aUfQGDdvJ1grEH7Q-adm9yCbAdSfcBYl8v64B7ABw8IRrJDA01vUwIYhxYvgl6Z9sAfSRtbsMV3MoxVBCtFtyt14UzFLQw"><img alt="From event streaming to datastore" src="https://blogger.googleusercontent.com/img/a/AVvXsEiGXcPDpLNk3o9NttWov4cTES-15DI78mzJ1dHJtM0aXkrCFx4PdDvpCExaZDvqwSr17K_Dgj5-P-71iOeSNHHK7BQ7YQ75aUfQGDdvJ1grEH7Q-adm9yCbAdSfcBYl8v64B7ABw8IRrJDA01vUwIYhxYvgl6Z9sAfSRtbsMV3MoxVBCtFtyt14UzFLQw=w640-h256" width="100%" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div></div></div>Before delving into the subject matter, the first question to ask is what is <i>idempotency</i>? A quick Google search provides us with the mathematical definition which is also relevant in terms of the software concept.</div><div><br /></div><div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgFW7pXR03VsktpTsAeS5COCkYiI-LObZAJ1JyZJ2jROqhvgFL9RFdTs9W7MHf-t7xdEmsOwkRCxfPK02QprFE4s0R5iqGV_2VUmwF4Eipk-yktM36DWpPhClZWlmu1A1MVoeC53CKXJTJ-pGpP8c4QKamgQDizlPcapmaDyiy6yaXEp_5RBUAT6FdjPQ" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1126" data-original-width="1980" height="365" src="https://blogger.googleusercontent.com/img/a/AVvXsEgFW7pXR03VsktpTsAeS5COCkYiI-LObZAJ1JyZJ2jROqhvgFL9RFdTs9W7MHf-t7xdEmsOwkRCxfPK02QprFE4s0R5iqGV_2VUmwF4Eipk-yktM36DWpPhClZWlmu1A1MVoeC53CKXJTJ-pGpP8c4QKamgQDizlPcapmaDyiy6yaXEp_5RBUAT6FdjPQ=w640-h365" width="640" /></a></div><div class="separator" style="clear: both; text-align: left;"><br /></div>The definition itself may be straightforward, however when a software solution involves consuming from an event stream and updating a record in a datastore, if idempotency is not considered as part of the development process, unexpected behaviours will certainly occur.</div><div><br /></div><div>Implying that each event being consumed will only be processed once in a distributed system is a fallacy. A very simple example can show one of many causes for data inconsistencies.</div><div><br /></div><div>Lets take a simplified example of storing an event stream's message into a relational database:</div><div><br /></div><div>
<pre><span style="color: blue;">while</span> (<span style="color: blue;">true</span>) {
<span style="color: blue;">var</span> message = consumeEventMessage();
db.upsertRecord(message);
acknowledgeEventMessage(message.Id);
}
</pre>
</div><div><br /></div><div>In this scenario, the database upsert could successfully complete and the subsequent acknowledgement of the message could fail which will result in a retrieval of the same message on the next run. If there is an aggregated field in the datastore, or a computation that occurs on-the-fly, the original data can be corrupted and result in erroneous data stored at rest. </div><div><br /></div><div>This wouldn't even be considered an edge case as it can happen more often than not. Network connections are never reliable and can drop out for a myriad of reasons. Building for failure and providing careful consideration for fault tolerance must be a part of any distributed system.</div><div><br /></div><div>As an example, in a given scenario where a system needs to cater for A/B test campaigns, the typical process would be as follows:</div><div><ol style="text-align: left;"><li>Set up an A/B test campaign to send to 1200 subscribers</li><li>Configure email A with a subject A and send it to 100 subscribers</li><li>Configure email B with a subject B and send it to 100 subscribers</li><li>After an hour, send the remaining 1000 subscribers to the email with the highest open rate</li></ol></div><div>From an event streaming perspective this is relatively straightforward:</div>
<div>
<pre>publish message { id: 1, email: 'A', subject: 'A', sent: 100, date: '2021-12-01 12:00:00' }
publish message { id: 2, email: 'B', subject: 'B', sent: 100, date: '2021-12-01 12:00:00' }
publish message { id: 1, email: 'A', subject: 'A', sent: 1000, date: '2021-12-01 13:00:00' }
</pre>
</div>
<div>Storing this into a normalised relational database table wouldn't be so difficult either:</div>
<div><br /></div>
<div class="table">
<table border="1" cellspacing="0" style="width: 100%; min-width: 475px;">
<tbody>
<tr style="background-color: #cfe2f3;">
<th style="padding: 10px;">id</th>
<th style="padding: 10px;">email</th>
<th style="padding: 10px;">subject</th>
<th style="padding: 10px;">date</th>
<th style="padding: 10px;">sent</th>
</tr>
<tr>
<td style="padding: 10px;">1</td>
<td style="padding: 10px;">A</td>
<td style="padding: 10px;">A</td>
<td align="right" style="padding: 10px;">2021-12-01 12:00:00</td>
<td align="right" style="padding: 10px;">100</td>
</tr>
<tr>
<td style="padding: 10px;">2</td>
<td style="padding: 10px;">B</td>
<td style="padding: 10px;">B</td>
<td align="right" style="padding: 10px;">2021-12-01 12:00:00</td>
<td align="right" style="padding: 10px;">100</td>
</tr>
</tbody></table>
</div>
<div><br /></div>
<div>Now with the upsert implementation <code>db.upsertRecord(message)</code> we can achieve parity with the total number of subscribers:</div>
<div><br /></div>
<div class="table">
<table border="1" cellspacing="0" style="width: 100%; min-width: 476px;">
<tbody>
<tr style="background-color: #cfe2f3;">
<th style="padding: 10px;">id</th>
<th style="padding: 10px;">email</th>
<th style="padding: 10px;">subject</th>
<th style="padding: 10px;">date</th>
<th style="padding: 10px;">sent</th>
</tr>
<tr>
<td style="padding: 10px;">1</td>
<td style="padding: 10px;">A</td>
<td style="padding: 10px;">A</td>
<td align="right" style="padding: 10px;">2021-12-01 13:00:00</td>
<td align="right" style="padding: 10px;">1100</td>
</tr>
<tr>
<td style="padding: 10px;">2</td>
<td style="padding: 10px;">B</td>
<td style="padding: 10px;">B</td>
<td align="right" style="padding: 10px;">2021-12-01 12:00:00</td>
<td align="right" style="padding: 10px;">100</td>
</tr>
</tbody></table>
</div>
<div><br /></div>
<div>In an ideal distributed world, this would work. But immediately we can see issues. The obvious one is the loss of data overwritten by the upsert, namely the initial sent date of the A/B test for email A. Another concern would be the fact that we seem to be calculating the total for a specific record on-the-fly and updating the sent count. The other issue isn't glaringly obvious unless a failure occurs in acknowledging the message as was mentioned above. If <code>acknowledgeEventMessage(message.Id)</code> is invoked for the third message and fails, we could get the following state after a retry:</div>
<div><br /></div>
<div class="table">
<table border="1" cellspacing="0" style="width: 100%; min-width: 478px;">
<tbody>
<tr style="background-color: #cfe2f3;">
<th style="padding: 10px;">id</th>
<th style="padding: 10px;">email</th>
<th style="padding: 10px;">subject</th>
<th style="padding: 10px;">date</th>
<th style="padding: 10px;">sent</th>
</tr>
<tr>
<td style="padding: 10px;">1</td>
<td style="padding: 10px;">A</td>
<td style="padding: 10px;">A</td>
<td align="right" style="padding: 10px;">2021-12-01 13:00:00</td>
<td align="right" style="padding: 10px;">2100</td>
</tr>
<tr>
<td style="padding: 10px;">2</td>
<td style="padding: 10px;">B</td>
<td style="padding: 10px;">B</td>
<td align="right" style="padding: 10px;">2021-12-01 12:00:00</td>
<td align="right" style="padding: 10px;">100</td>
</tr>
</tbody></table>
</div>
<div><br /></div>
<div>What is worse is that we have no way of undoing this unless we clear the database entirely and replay the entire event stream (or at a checkpoint) which would add a lot of development hours and maintenance overhead to resolve the issue.
</div>
<div><br /></div>
<div>In order to make the system idempotent for either the case of failure or a scenario where event streams would be replayed on existing data, the original data needs to be persisted in such a way that would still benefit the normalised state of the relational database table. The following schema can help achieve this:
</div>
<div><br /></div>
<div class="table">
<table border="1" cellspacing="0" style="width: 100%; min-width: 634px;">
<tbody>
<tr style="background-color: #cfe2f3;">
<th style="padding: 10px;">id</th>
<th style="padding: 10px;">email</th>
<th style="padding: 10px;">sub</th>
<th style="padding: 10px;">date_1</th>
<th style="padding: 10px;">sent_1</th>
<th style="padding: 10px;">date_2</th>
<th style="padding: 10px;">sent_2</th>
<th style="padding: 10px;">computed</th>
</tr>
<tr>
<td style="padding: 10px;">1</td>
<td style="padding: 10px;">A</td>
<td style="padding: 10px;">A</td>
<td align="right" style="padding: 10px;">...12:00</td>
<td align="right" style="padding: 10px;">100</td>
<td align="right" style="padding: 10px;">...13:00</td>
<td align="right" style="padding: 10px;">1000</td>
<td align="right" style="padding: 10px;">1100</td>
</tr>
<tr>
<td style="padding: 10px;">2</td>
<td style="padding: 10px;">B</td>
<td style="padding: 10px;">B</td>
<td align="right" style="padding: 10px;">...12:00</td>
<td align="right" style="padding: 10px;" align="right">100</td>
<td align="right" style="padding: 10px; color: #b7b7b7;">(null)</td>
<td align="right" style="padding: 10px; color: #b7b7b7;">(null)</td>
<td align="right" style="padding: 10px;">100</td>
</tr>
</tbody></table>
</div>
<div><br /></div>
<div>A change in table structure can help alleviate the problems of duplicate messages being consumed. It also enables persisting original data in order to avoid data loss. The unfortunate side effect however is that some records will not utilize all the data fields and will inevitably contain null values and that could be a deal breaker for the purists. Perhaps even normalizing the database table further into two separate tables with a one-to-many relationship would suffice, but that depends entirely on the volume of records and query efficiency surrounding the total sent count.
</div>
<div><br /></div>
<div>Moreover, events could be consumed out of order because there is no guarantee of order unless messages are consumed from the same partition. The above solution would still cater for this anomaly if sent_1 was 1000 and sent_2 was 100, and therefore our idempotent implementation is fulfilled.
</div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0tag:blogger.com,1999:blog-6898784454956902947.post-86550775531208101382020-02-10T13:57:00.000+00:002020-02-13T05:53:59.309+00:00Multiple Spark Streaming Jobs in a Single EMR Cluster<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKkn5Go3bsTy5HcY7CzqPrnvtfpCpo7MVgZItIiu4C7CTVg_QqCR1sTS1WImpBYPZWsDYUXXau5B0Yd71a_RJEJqlQpeewjOPYSNFRWbHWQqPSHjtYgn2BLGX2TU1h33ZIw1G2quFXQ_fO/s1600/emr.console.png"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjKkn5Go3bsTy5HcY7CzqPrnvtfpCpo7MVgZItIiu4C7CTVg_QqCR1sTS1WImpBYPZWsDYUXXau5B0Yd71a_RJEJqlQpeewjOPYSNFRWbHWQqPSHjtYgn2BLGX2TU1h33ZIw1G2quFXQ_fO/s1600/emr.console.png" width="100%" /></a></div>
<br />
A managed Spark cluster can always save time on infrastructure maintenance. <a href="https://aws.amazon.com/emr">EMR</a> (Elastic Map Reduce) is a tool that empowers data engineers to focus on building jobs to run on Spark rather than spending days managing the architecture of the cluster. However, that doesn't come without its shortfalls.<br />
<br />
As you may already know, you can only run one job at a time in an EMR cluster. This may not be too much of an issue when running a finite Spark job or batched process since there is a completion time, however that is not the case with Spark's <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html">Structured Streaming</a> because there is no end time!<br />
<br />
In a self-managed vanilla Spark cluster, it is possible to submit multiple jobs to a YARN resource manager and distribute the CPU and memory allocation to share its resources even when the jobs are running Structured Streaming.<br />
<br />
Unfortunately submitting a job to an EMR cluster that already has a job running will queue the newly submitted job. It will remain in a pending state until the previous job completes, which never does in a streaming scenario.<br />
<div class="separator">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitqS4viUrKt4Q7mD2daXIWQibTTqBh423XVoab7aZipZbTEoyJx2kGswNU5mBDcPpUIMNXYUt7jn6rNF4dvJ-BB3lTlY1DZ_CZmrub95LA7sDGwovD78eUFCIa-z0zE4BdHvTssaT-6nmB/s1600/emr.step.pending.png"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitqS4viUrKt4Q7mD2daXIWQibTTqBh423XVoab7aZipZbTEoyJx2kGswNU5mBDcPpUIMNXYUt7jn6rNF4dvJ-BB3lTlY1DZ_CZmrub95LA7sDGwovD78eUFCIa-z0zE4BdHvTssaT-6nmB/s640/emr.step.pending.png" width="100%" /></a></div>
Having one EMR cluster for every job wouldn't be the best solution however, because AWS infrastructure costs can grow pretty quickly, especially when consulting the <a href="https://aws.amazon.com/emr/pricing/" target="_blank">EMR pricing</a> page. The main cost issues revolve around having a master node created for every EMR cluster which only passes job requests to the core nodes, and those cores may not even have their resources fully utilized for smaller streaming tasks.<br />
<br />
This means rethinking the strategy of writing streaming jobs for use within an EMR cluster in order to work around this limitation. For this we need to look into improving the application layer.<br />
<h3 style="text-align: left;">
A Typical Structured Streaming Application</h3>
The primary reason a data engineer will require the use of Structured Streaming is for consuming data from an event source, transforming it based on specific business rules or for the purpose of aggregation, and then finally persisting the result of the transformation in a data store. Typically the Scala code would require three main steps.<br />
<br />
1. Reading from the event payload<br />
<pre>
<span style="color: orange;">def</span> <span style="color: #ffc66d;">getEmailOpenedEvent</span>(rawEvent: Array[<span style="color: orange;">Byte</span>]): <span style="color: #4e807d;">List</span>[EmailAggregateEvent] = {
<span style="background-color: yellow;">EmailOpened</span>.parseFrom(rawEvent)
.map(msg =>
EmailAggregateEvent(
msg.getEmailId.toByteArray,
msg.getSubscriberId.toByteArray,
msg.getOpenedDateUtc.toTimestamp
)
).toList
}
</pre>
Firstly, as shown in the above code, we will need the ability to parse the payload from the event stream. The <code>EmailOpened</code> type is generated based on a standard byte array format (e.g. <a href="https://github.com/protocolbuffers/protobuf">Protobuf</a>, <a href="https://avro.apache.org">Avro</a> or <a href="https://thrift.apache.org">Thrift</a>).<br />
<br />
2. Transforming the messages into aggregated hourly buckets<br />
<pre>
<span style="color: orange;">def</span> <span style="color: #ffc66d;">transformEmailOpened</span>(ds: Dataset[EmailAggregateEvent]): Dataset[EmailAggregateData] = {
<span style="color: orange;">import</span> ds.sparkSession.implicits._
ds.withWatermark(<span style="color: #9876aa;">"timestamp"</span>, <span style="color: #9876aa;">"24 hours"</span>)
.groupBy(
col(<span style="color: #9876aa;">"emailId"</span>),
window(col(<span style="color: #9876aa;">"timestamp"), "1 hour"</span>)
)
.agg(count(<span style="color: #9876aa;">"*"</span>).as(<span style="color: #9876aa;">"eventCount"</span>))
.withColumn(<span style="color: #9876aa;">"timestampBucket"</span>, col(<span style="color: #9876aa;">"window.start"</span>))
.as[EmailAggregateData]
}
</pre>
As shown above, a typical solution on how to work with streams is by aggregating the number of events based on a <code>timestamp</code> and maintaining an hourly bucket, which is then parsed as an <code>EmailAggregateData</code> type. For more information, the online programming guide on <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#window-operations-on-event-time">window operations in Structured Streaming</a> can provide further detail on this type of implementation.<br />
<br />
3. Starting the process<br />
<pre>
<span style="color: orange;">def</span> <span style="color: #ffc66d;">process</span>(topic: <span style="color: #4e807d;">String</span>, sql: <span style="color: #4e807d;">String</span>): <span style="color: orange;">Unit</span> = {
<span style="color: orange;">import</span> spark.implicits._
<span style="color: orange;">val</span> stream = SparkSession.builder.getOrCreate
.<span style="background-color: yellow;">readEventStream</span>(config.brokers, topic)
.flatMap(<i>getEmailOpenedEvent</i>(_))
<span style="color: orange;">val</span> query = <i>transformEmailOpened</i>(stream)
.writeStream
.foreach(<span style="background-color: yellow;">databaseWriter</span>(sql))
.trigger(Trigger.ProcessingTime(<span style="color: #9876aa;">"5 minutes"</span>))
.start
query.awaitTermination
}
</pre>
Lastly, we will need to start the process of consuming and writing to our data store. The <code>readEventStream()</code> function is an abstraction of <code>SparkSession.readStream()</code> to consume from an event source (either <a href="https://kafka.apache.org/">Apache Kafka</a>, <a href="https://aws.amazon.com/kinesis">Amazon Kinesis</a>, or <a href="https://azure.microsoft.com/en-au/services/event-hubs">Azure Event Hub</a> etc). The <code>databaseWriter()</code> function creates a JDBC connection to execute SQL upsert commands.<br />
<br />
There is an issue with this approach however.<br />
<br />
If it is not already obvious, this code can quickly grow. Similar code will need to be duplicated for every event to consume from and aggregate on. Moreover, this code would require separate EMR clusters, so the code will need to be consolidated.<br />
<h3 style="text-align: left;">
Consolidating repeated code using generics</h3>
Since Scala is flexible in providing a hybrid of both functional programming and Java's familiar object-oriented concepts, leveraging the use of base classes can provide great advantages.<br />
<pre><span style="color: orange;">class</span> Event <span style="color: orange;">extends</span> Product {
<span style="color: green;">// base functions</span>
}
<span style="color: orange;">case class</span> EmailAggregateEvent(
emailId: Array[<span style="color: orange;">Byte</span>],
subscriberId: Array[<span style="color: orange;">Byte</span>],
timestamp: Timestamp
) <span style="background-color: yellow;"><span style="color: orange;">extends</span> Event</span> {
<span style="color: green;">// override functions</span>
}
</pre>
There is one important note to consider about the above inheritance. Although Scala is flexible in enabling object-oriented programming techniques, we are still dealing with the caveats of Spark, namely serializable objects required for passing information between executors. Attempting to use a <code>trait</code> will actually throw an error:<br />
<pre><span style="color: red;">Error: scala.ScalaReflectionException: <none> is not a term</span>
</pre>
This is because it simply cannot serialize a non-concrete type, and the Spark runtime is not (yet) clever enough to understand inheritance, so it will attempt to serialize the actual trait itself, which is impossible.<br />
<br />
Moreover, when using a concrete class as the base class it will need to extend off the <code>Product</code> trait. This is because a <code>case class</code> automatically extends the <code>Product</code> trait which allows Spark to recognizes complex types at runtime for serialization.<br />
<br />
Inheritance will enable us to achieve the following:<br />
<pre><span style="color: orange;">case class</span> Job[<span style="color: #4e807d;">T</span> >: Event, <span style="color: #4e807d;">U</span> >: Data](
topic: <span style="color: #4e807d;">String</span>,
consume: Array[<span style="color: orange;">Byte</span>] => <span style="color: #4e807d;">List</span>[<span style="color: #4e807d;">T</span>],
transform: Dataset[<span style="color: #4e807d;">T</span>] => Dataset[<span style="color: #4e807d;">U</span>],
sql: <span style="color: #4e807d;">String</span>
)
</pre>
The <code>Job</code> type can now help substitute any job-specific implementation in the <code>process()</code> function as follows:<br />
<pre><span style="color: orange;">def</span> <span style="color: #ffc66d;">process</span>[<span style="color: #4e807d;">T</span> >: Event, <span style="color: #4e807d;">U</span> >: Data](<span style="background-color: yellow;">job</span>: Job[<span style="color: #4e807d;">T</span>, <span style="color: #4e807d;">U</span>]): <span style="color: orange;">Unit</span> = {
<span style="color: orange;">import</span> spark.implicits._
<span style="color: orange;">val</span> stream = SparkSession.builder.getOrCreate
.readEventStream(config.brokers, <span style="background-color: yellow;">job</span>.topic)
.flatMap(<span style="background-color: yellow;">job</span>.consume(_))
<span style="color: orange;">val</span> query = <span style="background-color: yellow;">job</span>.transform(stream)
.writeStream
.foreach(databaseWriter(<span style="background-color: yellow;">job</span>.sql))
.trigger(Trigger.ProcessingTime(<span style="color: #9876aa;">"5 minutes"</span>)))
.start
query.awaitTermination
}
</pre>
Unfortunately this will still not be enough.<br />
<br />
In the original <code>transformEmailOpened()</code> function above, there are implicit conversions which can no longer work when using inheritance. The Spark API provides an easy to use property called <code>as[T]</code> which returns a new Dataset where each field is mapped to columns of the same name of the given type <code>T</code>. During runtime Spark will need a little help to understand what type is required explicitly, so the <code>transformEmailOpened()</code> function will need to change as follows:<br />
<pre><span style="color: orange;">def</span> <span style="color: #ffc66d;">transformEmailOpened</span>(ds: Dataset[EmailAggregateEvent]): Dataset[EmailAggregateData] = {
<span style="color: orange;">import</span> ds.sparkSession.implicits._
ds
<span style="color: green;">// explicitly map to concrete type</span>
<span style="background-color: yellow;">.map(row => EmailAggregateEvent(</span>
<span style="background-color: yellow;">row.emailId,</span>
<span style="background-color: yellow;">row.subscriberId,</span>
<span style="background-color: yellow;">row.timestamp</span>
<span style="background-color: yellow;">))</span>
.withWatermark(<span style="color: #9876aa;">"timestamp"</span>, <span style="color: #9876aa;">"24 hours"</span>)
.groupBy(
col(<span style="color: #9876aa;">"emailId"</span>),
window(col(<span style="color: #9876aa;">"timestamp"</span>), <span style="color: #9876aa;">"1 hour"</span>)
)
.agg(count(<span style="color: #9876aa;">"*"</span>).as(<span style="color: #9876aa;">"eventCount"</span>))
.withColumn(<span style="color: #9876aa;">"timestampBucket"</span>, col(<span style="color: #9876aa;">"window.start"</span>))
<span style="color: green;">// replace as[T] with explicit map</span>
<span style="background-color: yellow;">.map(row => EmailAggregateData(</span>
<span style="background-color: yellow;">row.getAs(<span style="color: #9876aa;">"emailId"</span>),</span>
<span style="background-color: yellow;">row.getAs(<span style="color: #9876aa;">"timestampBucket"</span>),</span>
<span style="background-color: yellow;">row.getAs(<span style="color: #9876aa;">"eventCount"</span>)</span>
<span style="background-color: yellow;">))</span>
}
</pre>
Inheritance can now work at runtime, and will enable our code to scale when more events are required for consumption. Adding multiple jobs can now be relatively easy.<br />
<br />
However, we cannot simply just write a <code>jobs.foreach()</code> implementation and expect it to work since it will never iterate past the first streaming job. This is where concurrency plays an important role in the overall solution.<br />
<h3 style="text-align: left;">
Concurrency</h3>
Enabling parallel programming in Scala is very simple. The <code>.par()</code> function enables parallel processing of a collection in an extremely concise way. We can take advantage of this by wrapping the entire functionality of the <code>process()</code> function into a parallel <code>foreach()</code> loop as follows:<br />
<pre>jobs.<span style="background-color: yellow;">par</span>.foreach(job => {
process(job)
})
</pre>
There is one thing that hasn't been considered with the above code though. The <code>SparkSession</code> that is used within the <code>process()</code> function is a static object and only one can be active per Spark cluster. However that doesn't mean we cannot start multiple streaming jobs using the same <code>SparkSession</code>! The <code>SparkSession.readEventStream()</code> functionality won't be a problem running in parallel even though the function is being called multiple times against the same static Spark session.<br />
<br />
In terms of concurrency though, the <code>.par()</code> functionality is a multi-threaded implementation. In the JVM framework each thread runs on one CPU unit (either a physical processor or hyper-threaded processor). This means a multi-threaded approach would only work if the number of jobs required is less than the number of processors in the EMR cluster. To find out how many cores (and essentially how many separate threads the job can run on) the following value can be outputted:<br />
<pre>Runtime.getRuntime.availableProcessors</pre>
The problem with this solution is cost. When considering that each job may only utilize 5% or less of the CPU units, we may have more cores in the EMR cluster than is necessary. As an example, running 12 jobs concurrently means the next instance size up to support this is <code>m4.4xlarge</code> which has 16 cores. That is a total of 4 cores that will sit idle never to be used. Moreover, each used core may also be underutilized.<br />
<br />
Considering asynchronous programming rather than a multi-threaded approach suddenly becomes more relevant, and Scala's <code>Futures</code> can come in handy with only a few simple changes:<br />
<pre><span style="color: orange;">val</span> executionContext = ExecutionContext.fromExecutorService(Executors.newFixedThreadPool(maxConcurrentJobs))
<span style="color: orange;">val</span> tasks: <span style="color: #4e807d;">List</span>[Future[<span style="color: orange;">Unit</span>]] = jobs
.map(job =>
<span style="background-color: yellow;">Future</span> {
process(job)
}(executionContext)
)
Await.result(Future.sequence(tasks)(implicitly, executionContext), Duration.Inf)
</pre>
With this minimal code change we can have massive benefits in cost reduction. By default the thread pool count (i.e. the maximum number of concurrent runs) is actually the number of processors, just like the multi-threading approach. This is where the <code>executionContext</code> comes in handy to explicitly set the number of concurrent executions with the <code>maxConcurrentJobs</code> variable. This number can be as large as necessary to handle as many job across the entire EMR cluster regardless of how many cores are provisioned. The limit can now be based on the exact number of cores and the resource allocation required per individual job.<br />
<h3 style="text-align: left;">
Conclusion</h3>
Even with the limitations of EMR clusters allowing only one job to run at any given time, as well as considering that only one static <code>SparkSession</code> object exists per Spark job, there are still efficient ways to work around these limitations.<br />
<br />
Jobs can be refactored to enable consolidation of code repetition which in turn enables the possibilities of concurrent processing. This can significantly reduce cost of infrastructure and keep a clean code base that can be extensible.<br />
<br /></div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0tag:blogger.com,1999:blog-6898784454956902947.post-19200142735117452712019-01-10T01:30:00.000+00:002020-02-10T09:32:49.808+00:00Compiling Spark Jobs using SBT on Build Machines<div dir="ltr" style="text-align: left;" trbidi="on">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5THJ9AlTCDUGP34DNmtafuDpRa-u2NKr9peGswubAvonf0_EOeyO-LBmtPC60tuh_qGuNxiOIYKKeHycL6O3oXfr7mhc-iTkSVlDhUhhRRq_0-0RJMYj-YOSSS-RXAakfEQQBb_MmL3oE/s1600/Continuous-Delivery-and-Deployment.jpg" imageanchor="1" >
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh5THJ9AlTCDUGP34DNmtafuDpRa-u2NKr9peGswubAvonf0_EOeyO-LBmtPC60tuh_qGuNxiOIYKKeHycL6O3oXfr7mhc-iTkSVlDhUhhRRq_0-0RJMYj-YOSSS-RXAakfEQQBb_MmL3oE/s1600/Continuous-Delivery-and-Deployment.jpg" width="100%" />
</a>
Deploying a Spark job can be challenging, especially considering that one Spark job is never the same as another. Deployment times can vary significantly depending on a wide range of factors. Finding ways to make the deployment efficient can greatly improve the process, and that can be achieved with a few simple strategies.<br /><br />
Consider the following steps required to deploy a Spark job that is built in <code>scala</code> and compiled using <code>sbt</code>:
<ol>
<li>Get latest project files <i>(requires <code>git</code> or <code>subversion</code> etc.)</i></li>
<li>Download dependencies <i>(requires <code>sbt</code>)</i></li>
<li>Compile Spark application <i>(requires <code>sbt</code>)</i></li>
<li>Run unit test <i>(requires <code>sbt</code>)</i></li>
<li>Package Spark application to a .jar file <i>(requires <code>sbt</code>)</i></li>
<li>Upload .jar file to an accessible location (e.g. s3 bucket) <i>(requires <code>aws-cli</code>)</i></li>
<li>Spark submit using the s3 location to the master node <i>(requires <code>spark</code>)</i></li>
</ol>
This can be achieved using a deployment pipeline created in a CI tool like TeamCity or Jenkins.<br />
Usually build machines used for a deployment pipeline are provisioned so they are lightweight with very few applications installed.<br />
In fact, if the only application installed on a build machine is docker, then that is enough.<br /><br />
Since our build machines don't have sbt installed, a docker image with <code>scala</code> and <code>sbt</code> is required to run <code>sbt assembly</code>. A docker image found on docker hub like <a href="https://hub.docker.com/r/spikerlabs/scala-sbt/">spikerlabs/scala-sbt</a> can achieve this. However, running <code>sbt assembly</code> using this docker image will take a significantly long time to complete, sometimes as long as 30 minutes! This is because all the necessary dependencies for your spark job will need to be downloaded before compiling your Spark application.<br /><br />
Performing this operation every time you need to deploy your Spark job is costly. So in order to improve the efficiency of builds to prevent these download times, the dependencies first need to be downloaded onto a specific sbt docker image that is tailored for your Spark job. This can then be used as part of the deployment pipeline.<br /><br />
The following steps will need to be carried out for this to be achieved:<br /><br />
1. <code>cd</code> into the project folder.<br /><br />
2. Run the following docker command to start a <code>spikerlabs/scala-sbt</code> container in interactive mode.<br />
<pre>
docker run -i -v "/$(pwd)/":/app -w "//app" --name "my-scala-sbt-container" "spikerlabs/scala-sbt:scala-2.11.8-sbt-0.13.15" bash
</pre>
Note that a specific <code>scala</code> and <code>sbt</code> version will alway be required otherwise the "latest" tag could fail during compilation with any breaking changes.<br /><br />
3. Once in interactive mode within the docker container, run the following commands within the container.
<pre>
> sbt assembly # to download dependencies, compile, and package the spark job
> exit # once the packaging is complete
</pre>
4. This will create a docker container called <code>my-scala-sbt-container</code> which will need to be exported, then imported as an image, as follows:
<pre>
docker export --output="./my-scala-sbt-container.tar" "my-scala-sbt-container"
docker import "./my-scala-sbt-container.tar" [INTERNAL_REPOSITORY_URL]/my-scala-sbt-image:[VERSION_NUMBER]
</pre>
Where <code style="font-size:12px">[INTERNAL_REPOSITORY_URL]</code> is a company wide docker repository location e.g. like <a href="https://www.sonatype.com/nexus-repository-sonatype">nexus</a>,<br />
and where <code style="font-size:12px">[VERSION_NUMBER]</code> needs to be bumped up from a previous version.<br /><br />
Note that the import allows a container that has all the necessary dependencies to be converted into an image called <code style="font-size:12px">[INTERNAL_REPOSITORY_URL]/my-scala-sbt-image:[VERSION_NUMBER]</code>.<br /><br />
This will be needed when <code>docker push</code> is executed. Unfortunately the docker API for a <code>push</code> command requires the image <i>name</i> to be exactly the same as the <i>docker repository url</i> which seems a little non-intuitive.<br /><br />
5. To publish the the local docker image to the internal docker repository, run the following <code>docker push</code> command:
<pre>
docker image push [INTERNAL_REPOSITORY_URL]/my-scala-sbt-image:[VERSION_NUMBER]
</pre>
Where <code style="font-size:12px">[INTERNAL_REPOSITORY_URL]</code> is a company wide docker repository location e.g. like <a href="https://www.sonatype.com/nexus-repository-sonatype">nexus</a>,<br />
and where <code style="font-size:12px">[VERSION_NUMBER]</code> is the same as the previous step.<br /><br />
This can now help in the CI deployment pipeline, where the step to run <code>sbt assembly</code> can be done with the <code>docker run</code> command as follows:
<pre>cat <<EOF <span style="color: orange;">> assembly.sh
<br />
#!/usr/bin/env bash
<br />
set -ex
<br />
<span style="background-color: yellow;">sbt assembly</span>
</span>
EOF
docker run \
--rm \
-v "/$(pwd)/..":/app \
-v "/$(pwd)/assembly.sh":/assembly.sh \
-w "//" \
--entrypoint=sh \
<span style="background-color: yellow;">[INTERNAL_REPOSITORY_URL]/my-scala-sbt-image:[VERSION_NUMBER]</span> \
//assembly.sh
</pre>
That should improve the build timings by removing up to 30 minutes off the deployment pipeline, depending on how many dependencies are required for the Spark application.
</div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0tag:blogger.com,1999:blog-6898784454956902947.post-53405900521757389302018-09-26T08:00:00.000+00:002020-02-10T09:34:15.127+00:00Automating Deployments for Spark Streaming on AWS EMR<div dir="ltr" style="text-align: left;" trbidi="on">
Anyone who's familiar with submitting a Spark job will know it's as simple as running a single command, so long as the Spark CLI is installed on the machine running that command:<br />
<pre>spark-submit --master ${MASTER_ENDPOINT} --deploy-mode cluster --class com.cm.spark.SparkApp s3://bucket/jobs/${APP_JAR}
</pre>
<div style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;">
<img alt="Elastic Map Reduce" src="https://d117h1jjiq768j.cloudfront.net/images/default-source/Products/datadirect/data-sources/amazon_emr/amazon_emr.png" width="145px" />
</div>
<br />
This is easy enough when working with a self-managed vanilla Spark cluster. AWS also provides a service called EMR (Elastic Map-Reduce) which offers hosting of Spark clusters and so the AWS CLI provides an equally effortless way to submit Spark jobs:<br />
<pre>aws emr add-steps cluster-id j-UJODR7SZ6L7L steps Type=Spark,Name="Aggregator",ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,--class,com.cm.spark.SparkApp,s3://bucket/jobs/${APP_JAR}]
</pre>
This is essentially a wrapper around the <code>spark-submit</code> command. The <a href="https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-submit-step.html">AWS documentation</a> is relatively detailed in providing information for adding additional parameters to the command line arguments in order to provide all the features found in the <code>spark-submit</code> command.
This makes it incredibly simple to create a deployment step in your CI pipeline.<br />
<br />
Even if the build machines don't have AWS CLI installed, the commands can be wrapped around a <code>docker run</code> call using an image called <code>garland/aws-cli-docker</code> which can be found on <a href="https://hub.docker.com/r/garland/aws-cli-docker/">Docker Hub</a>:
<br />
<pre>cat <<EOF <span style="color: orange;">> submit.sh
#!/usr/bin/env bash
set -ex
<span style="background-color: yellow;">aws emr add-steps cluster-id j-UJODR7SZ6L7L steps ...</span>
</span>
EOF
docker run \
--rm \
-e AWS_DEFAULT_REGION=${AWS_REGION} \
-e AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
-e AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
-v "/${APP_JAR}":/${APP_JAR} \
-v "/$(pwd)/submit.sh":/submit.sh \
-w "//" \
--entrypoint=sh \
<span style="background-color: yellow;">garland/aws-cli-docker:latest</span> \
//submit.sh
</pre>
The beautiful thing about docker is that the build machines only ever need one prerequisite: docker. Anything else that needs to be installed for deployment can instead be found as a docker image on <a href="https://hub.docker.com/">Docker Hub</a>. In this case the <code>aws emr add-steps</code> command is executed within a docker container and the build machines don't need to know anything about it!<br />
<br />
However, there is a catch. This is only straightforward if you are under the assumption that your Spark job eventually completes, whether it returns a <b>success</b> or <b>failure</b> status. In fact, that is exactly the same assumption that the AWS documentation makes. Once a job completes, running the same command will add a new step on the cluster and the step will run without any issues.<br />
<br />
But what about <i>Structured Streaming</i>? An ongoing Spark job with an open connection to listen to an event stream suddenly makes one simple command slightly more complicated when deployment of a code change is required.<br />
<br />
This would require stopping a currently running step before adding a new step. <a href="https://docs.aws.amazon.com/cli/latest/reference/emr/cancel-steps.html">AWS documentation</a> has another command that can be useful in this scenario:<br />
<pre>aws emr cancel-steps --cluster-id j-UJODR7SZ6L7L --step-ids s-2V57YI7T5NF42
</pre>
Unfortunately the <code>cancel-steps</code> command can only remove a <b>pending</b> step i.e. one that isn't in a <b>running</b> state. In fact there is no API to terminate a running step at all and the only solution found in <a href="https://aws.amazon.com/premiumsupport/knowledge-center/cancel-emr-step/">AWS documentation</a> is to do the following:<br />
<ol>
<li>Log into the EMR master node (which will need a secure .pem file)</li>
<li>Run <b><code>yarn application -list</code></b></li>
<li>Find the application ID of the currently running <b>job</b></li>
<li>Run <b><code>yarn application -kill ${APP_ID}</code></b></li>
</ol>
Obviously this is not a viable solution for automation since we will have to deal with ssh and passing .pem files around on build machines in order to achieve this process. No doubt your security team would have their own opinion on this approach.<br />
<br />
Alternatively, there is another <a href="https://docs.aws.amazon.com/cli/latest/reference/emr/terminate-clusters.html">AWS API call</a> that may be a useful workaround:<br />
<pre>aws emr terminate-clusters --cluster-ids j-UJODR7SZ6L7L
</pre>
This could achieve what we are looking for, if the following automation steps are executed:<br />
<ol>
<li>Terminate EMR cluster</li>
<li>Recreate EMR cluster using terraform or cloud formation</li>
<li>Add step to newly created cluster</li>
</ol>
Terminating an entire cluster just to deploy a job may seem like overkill, however in the new world of infrastructure-as-code, it isn't quite as insane as one might think. After consulting AWS experts and Spark specialists alike, both have agreed that these days it is actually encouraged to dedicate one spark job to a single Spark cluster.<br />
<br />
The benefits are quite significant:
<br />
<ul>
<li>Dedicated clusters mean each job can only affect itself without consuming resources from other jobs</li>
<li>If there are any fatal errors or memory leaks other jobs aren't affected</li>
<li>The entire cluster's resources can be specifically tailored for a single job, rather than maintaining resource allocation for many jobs which will require significant monitoring and any necessary resizing will affect all currently running jobs with downtime</li>
</ul>
However, the downfalls are also notable:
<br />
<ul>
<li>Rather than one single master with many shared cores, there will be a master for each job which will obviously mean more EC2 costs, <i>however</i> masters can be kept small since they're only required to receive requests and distribute compute to slaves</li>
<li>Deployment pipelines will be slower since recreating clusters for every application code update will add significant time to the build process</li>
</ul>
These downfalls may be deal breakers for some use cases, and it has given enough attention to the AWS experts after consultation to at least rethink the API strategies they've provided for a scenario such as Structured Streaming. The initial assumption that a Spark job is a short lived process is simply not enough.<br />
<br />
A better solution for now would actually be to utilise the YARN API instead. Since YARN is the orchestrator of the Spark nodes within the cluster, it provides many features in which to manage both the cluster and its jobs. There is <a href="https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">comprehensive documentation</a> on every command available via HTTP REST API calls.
That means the build machines can avoid having to ssh into master nodes using .pem files and can send an HTTP request instead (so long as the inbound firewall permissions of the Spark cluster allow the build machines access to it - and that is a simple terraform or cloud formation configuration).<br />
<br />
That means the following automation steps can achieve a successful deployment pipeline:
<br />
<pre><span style="color: green;"># 1. Get a list of active jobs</span>
APPS=<span style="color: orange;">$(curl -s -X GET -H "Content-Type: application/json" "http://${MASTER_ENDPOINT}/ws/v1/cluster/apps")</span>
<span style="color: green;"># 2. Parse the response payload and return the id of the running job (requires jp package)</span>
APP_ID=<span style="color: orange;">$(echo $APPS | jq '.apps.app[0].id')</span>
<span style="color: green;"># 3. Send a kill request to the currently running job</span>
curl -X PUT -H "Content-Type: application/json" -d '{"state": "KILLED"}' "http://${MASTER_ENDPOINT}/ws/v1/cluster/apps/${APP_ID}/state"
<span style="color: green;"># 4. Add a new step to submit the new job</span>
aws emr add-steps cluster-id j-UJODR7SZ6L7L steps Type=Spark,Name="Aggregator",ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,--class,com.cm.spark.SparkApp,s3://bucket/jobs/${APP_JAR}]
</pre>
This removes all the downfalls because it creates a separation of concerns by providing a specific CI pipeline for job deployment while the cluster can be resized and rebuilt when necessary. The benefits of keeping one dedicated cluster per Spark job can still be maintained with this solution, only that the cluster doesn't have to be destroyed every time there is an iteration required on the job itself.<br />
<br /></div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com1tag:blogger.com,1999:blog-6898784454956902947.post-80236529598515743102017-10-30T05:46:00.002+00:002018-09-28T16:29:28.842+00:00Aggregating Event Streams in Redshift using Matillion<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="graf graf--p" name="1a3b">
<div class="graf graf--p" name="1a3b">
If you haven’t heard of <a class="markup--anchor markup--p-anchor" data-href="https://www.matillion.com/" href="https://www.matillion.com/" rel="noopener" target="_blank">Matillion</a>, then it’s time you knew about it. If you’ve used ETL tools in the past like the traditional SSIS for Microsoft’s SQL Server, then Matillion would be very familiar to you. Where SSIS was tailored for SQL Server, Matillion is tailored for Amazon’s Data Warehouse offering, AWS Redshift.</div>
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibmCN3BBMlgK-zrJP3ZqBbOzoqzrsiOkDUxUUf3je6kWubxy-3jYEElEOdA4uWTuOqpOxXphLQdLFLSxBlGKRgkYvwftNBALG_w3EADm8sURoauyA0vwwk_DDVqY2_Eh1yIkPRLXhkjzBC/s1600/etl.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="427" data-original-width="1189" height="227" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibmCN3BBMlgK-zrJP3ZqBbOzoqzrsiOkDUxUUf3je6kWubxy-3jYEElEOdA4uWTuOqpOxXphLQdLFLSxBlGKRgkYvwftNBALG_w3EADm8sURoauyA0vwwk_DDVqY2_Eh1yIkPRLXhkjzBC/s640/etl.png" width="640" /></a></div>
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0rRy22bvye1H9SX66SfZrUZRqjF4KEtaoQwYVKwrbaYo_jikWHt6uSgDnpmCMnF0084ogWyrQUhg48b4Qj0dO7qTijCVKZOzLz2oDly-rtSiyLcdkb1JLoXUOaS2jqgNAP8Swm51XItb_/s1600/extract.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="226" data-original-width="173" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh0rRy22bvye1H9SX66SfZrUZRqjF4KEtaoQwYVKwrbaYo_jikWHt6uSgDnpmCMnF0084ogWyrQUhg48b4Qj0dO7qTijCVKZOzLz2oDly-rtSiyLcdkb1JLoXUOaS2jqgNAP8Swm51XItb_/s1600/extract.png" /></a><br />
<div class="graf graf--p" name="d9b0">
Matillion’s user interface is very intuitive as it groups all its components into three natural categories: those for <strong class="markup--strong markup--p-strong"><em class="markup--em markup--p-em">extract</em></strong>, those for <strong class="markup--strong markup--p-strong"><em class="markup--em markup--p-em">transform</em></strong>, and those for <strong class="markup--strong markup--p-strong"><em class="markup--em markup--p-em">load</em></strong>.</div>
<br />
<div class="graf graf--p" name="5fd7">
Since Matillion sits on top of Redshift, both must live within AWS, so the features for <em class="markup--em markup--p-em">extracting</em> include components for RDS (as well as any other relational databases that support JDBC drivers). It also supports DynamoDB, S3, as well as components for MongoDB, Cassandra and Couchbase.</div>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYHIGJSnNkNkJK7UrANaXzQkrUa8TJ7J1fNWkVyCRl5BbdtwQYnahb3oD05ysj011nZpGouuSubTlyNqcYGQIw74ySFh64ECbiJFUYOev-j0tF_6byUZpA92egahtFwlk7egH35TVCT4zw/s1600/transform.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="249" data-original-width="181" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYHIGJSnNkNkJK7UrANaXzQkrUa8TJ7J1fNWkVyCRl5BbdtwQYnahb3oD05ysj011nZpGouuSubTlyNqcYGQIw74ySFh64ECbiJFUYOev-j0tF_6byUZpA92egahtFwlk7egH35TVCT4zw/s1600/transform.png" /></a><br />
<div class="graf graf--p" name="7305">
Components used for <em class="markup--em markup--p-em">transformation</em> provide the ability to aggregate, filter, rank, and even to roll up sums over multiple records from a transient “staging” Redshift table to <em class="markup--em markup--p-em">load</em> it to a more permanent Redshift table. The underlying functions for these components are in fact Redshift query commands.</div>
<br />
<div class="graf graf--p" name="22da">
Matillion seems quite mature with countless component features at your fingertips. Much of the transformation logic you can construct can definitely be written by hand using Redshift’s comprehensive <a class="markup--anchor markup--p-anchor" data-href="http://docs.aws.amazon.com/redshift/latest/dg/c_SQL_commands.html" href="http://docs.aws.amazon.com/redshift/latest/dg/c_SQL_commands.html" rel="noopener" target="_blank">documentation on ANSI SQL commands</a>, but what Matillion can provide is a visual representation which significantly improves collaboration amongst team members so they can all be equally responsible for maintaining the ETL process in an efficient way.</div>
<br />
<div class="graf graf--p" name="f4c1">
Many could be forgiven for associating an ETL process with the old techniques of extracting large amounts of data from a monolithic data store to push into a read store using the typical CQRS architecture, but Matillion can be used for much more than that.</div>
<br />
<div class="graf graf--p" name="c69e">
Our use-case at <a class="markup--anchor markup--p-anchor" data-href="https://www.campaignmonitor.com/" href="https://www.campaignmonitor.com/" rel="noopener" target="_blank">Campaign Monitor</a> not only required data from a snapshot-style structure as is found in a traditional monolithic relational database, but also data that is sourced from an event stream for a rolling daily aggregation.</div>
<br />
<div class="graf graf--p" name="beee">
In order to achieve this we needed an initial query to obtain the total snapshot on a particular day, which can then be stored in a “roll-up” table in Redshift as the first day of aggregated totals. Let’s take the following table as an example snapshot that can be the baseline for any future events:</div>
<br />
<table border="1" cellspacing="0" style="width: 100%;">
<tbody>
<tr><th>ListID</th><th>Subscribers</th><th>Date</th></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>1</td><td>1000</td><td>2017-10-01</td></tr>
<tr align="right"><td>2</td><td>2000</td><td>2017-10-01</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>3</td><td>3000</td><td>2017-10-01</td></tr>
<tr align="right"><td>4</td><td>4000</td><td>2017-10-01</td></tr>
</tbody></table>
<br />
<div class="graf graf--p" name="52d7">
This can be done using a traditional ETL job in Matillion as follows:</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuRmxxyRO7hSsNtUuc67igDrGLwyn8xElt7zcTXLqiyPHr9q4KQXCUabIB756VY054noI8H5bGjlADzjVr1NG2RpG4nVg9VQwldFmTGt8LB0KhoMl9msbdHW5T2GqFggSHyv67GKLSLu40/s1600/snapshot.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="271" data-original-width="602" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuRmxxyRO7hSsNtUuc67igDrGLwyn8xElt7zcTXLqiyPHr9q4KQXCUabIB756VY054noI8H5bGjlADzjVr1NG2RpG4nVg9VQwldFmTGt8LB0KhoMl9msbdHW5T2GqFggSHyv67GKLSLu40/s1600/snapshot.png" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="graf graf--p" name="2d0a">
There is a lot of sophisticated magic under the hood with regards to the <em class="markup--em markup--p-em">Database Query </em>component. The SQL query that is run from the relational database will return a resultset that Matillion can chunk into separate concurrent processes. Each of these processes will create an underlying S3 file before loading the data into the “rs_lists_subscriber_rollup_staging” table in Redshift.</div>
<br />
<div class="graf graf--p" name="1929">
Once we have a snapshot on a particular date (in this case 2017–10–01 in the example above) all events for subscribers that come in after this date can be streamed from our event source, and then aggregated into the same table with a rolling count of subscribers for subsequent dates.</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="graf graf--p" name="f88c">
This requires some form of consumption externally to receive the incoming events, which can be batched into S3 files, and then persisted into an ephemeral “raw” table in Redshift using the COPY command. This raw table would look as follows:</div>
<br />
<div class="table">
<table border="1" cellspacing="0" style="min-width: 350px; width: 100%;"><tbody>
<tr><th>ListID</th><th>SubscriberID</th><th>Subscribe Tally</th><th>Date</th></tr>
<tr align="right"><td>1</td><td>1</td><td>1</td><td>2017-10-02</td></tr>
<tr align="right"><td>1</td><td>2</td><td>-1</td><td>2017-10-02</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>3</td><td>1</td><td>1</td><td>2017-10-02</td></tr>
<tr align="right"><td>2</td><td>1</td><td>1</td><td>2017-10-02</td></tr>
<tr align="right"><td>2</td><td>2</td><td>1</td><td>2017-10-02</td></tr>
<tr align="right"><td>2</td><td>3</td><td>1</td><td>2017-10-02</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>4</td><td>1</td><td>-1</td><td>2017-10-02</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>4</td><td>2</td><td>-1</td><td>2017-10-02</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>4</td><td>3</td><td>-1</td><td>2017-10-02</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>4</td><td>4</td><td>1</td><td>2017-10-02</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>4</td><td>1</td><td>1</td><td>2017-10-03</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>4</td><td>5</td><td>1</td><td>2017-10-03</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>4</td><td>6</td><td>1</td><td>2017-10-03</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>4</td><td>7</td><td>1</td><td>2017-10-03</td></tr>
</tbody></table></div>
<br />
<div class="graf graf--p" name="e207">
Generally, a contact subscribes or unsubscribes at a specific time from a list, and this will need to be aggregated daily per list so the data above would look as follows:</div>
<br />
<div class="table">
<table border="1" cellspacing="0" style="width: 100%;"><tbody>
<tr><th>ListID</th><th>Subscribers</th><th>Date</th></tr>
<tr align="right"><td>1</td><td>0</td><td>2017-10-02</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>2</td><td>3</td><td>2017-10-02</td></tr>
<tr align="right"><td>3</td><td>1</td><td>2017-10-02</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>4</td><td>-2</td><td>2017-10-02</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>4</td><td>4</td><td>2017-10-03</td></tr>
</tbody></table></div>
<br />
This aggregation can also be done in Matillion very easily from the raw table:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh30FTcvTYVQvKEh6UjgHMAH30i_m8vwVUn5d1HIfdugrGGfO0hwOzqoYyha8E1twTamGb5VTeCcP2Y2y8LCHCT1ixVtbzVmO9eqCCtwWa04wCgXV3mqi2szQ3cX5_tOSScVQ58lXBCY9DN/s1600/aggregation.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="155" data-original-width="448" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh30FTcvTYVQvKEh6UjgHMAH30i_m8vwVUn5d1HIfdugrGGfO0hwOzqoYyha8E1twTamGb5VTeCcP2Y2y8LCHCT1ixVtbzVmO9eqCCtwWa04wCgXV3mqi2szQ3cX5_tOSScVQ58lXBCY9DN/s1600/aggregation.png" /></a></div>
<br />
<div class="graf graf--p" name="c164">
The <em class="markup--em markup--p-em">Aggregate</em> component would simply need to sum the Subscribe/Unsubscribe column with a grouping of ListID. Once this is achieved, there will need to be a union of data from the existing snapshot data so that there can be a baseline to work with for rolling counts. Only then can the very powerful <em class="markup--em markup--p-em">Window Calculation</em> component be used for rolling up the sum of subscribers based on a partition (in this case the ListID) and a partition ordering (in this case Date). The Matillion job would look as follows:</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigx9eBcHykwvBO_BX2YkpYriLNaArHevVNhRcqByjZAcR0-K5C5WE4qrnV7pF4H7-na1_Rke3qfZskVQ-6gt-omb8hcQtUj_udY6fqeEnU2t3-XKFiOrVLs2LbeS_fdKbGhlxUdTL2LQpt/s1600/window.calculation.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="306" data-original-width="849" height="230" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigx9eBcHykwvBO_BX2YkpYriLNaArHevVNhRcqByjZAcR0-K5C5WE4qrnV7pF4H7-na1_Rke3qfZskVQ-6gt-omb8hcQtUj_udY6fqeEnU2t3-XKFiOrVLs2LbeS_fdKbGhlxUdTL2LQpt/s640/window.calculation.png" width="640" /></a></div>
<br />
<div class="graf graf--p" name="a75f">
There are further properties for the <em class="markup--em markup--p-em">Windows Calculation</em> component that need to be considered, including setting the lower bound to “unbounded preceding”. This ensures that the calculated sum is rolled up from the very first record of the dataset that was unioned from the previous component. The properties would look as follows:</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5jwv2mORyeu5YwRQJSEVPJ7c_Awf7_1SchRMh5uEL5FmxNSGJ6lwrbg_oi2nXVOl8PDPEHltoAQegXpo0nl9N4OuwmsOkmi8JMtktzGhJkMB9E6u_sU_h_JfhGbNwjcGVx3re-rM2VinW/s1600/window.calculation.properties.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="293" data-original-width="543" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5jwv2mORyeu5YwRQJSEVPJ7c_Awf7_1SchRMh5uEL5FmxNSGJ6lwrbg_oi2nXVOl8PDPEHltoAQegXpo0nl9N4OuwmsOkmi8JMtktzGhJkMB9E6u_sU_h_JfhGbNwjcGVx3re-rM2VinW/s1600/window.calculation.properties.png" /></a></div>
<br />
<div class="graf graf--p" name="f207">
There are of course other edge cases that will need to be considered that could slowly make your transformation job increasingly complex. These can include lists that were created after the snapshot was taken, or even more interesting is catering for missing days that receive no events which will need to be populated with a default of zero so aggregation can be included in the roll-up for that day. Thankfully Matillion makes it simple to grow on the above transformation so it can be easily maintained.</div>
<br />
The final table would then look as follows:<br />
<br />
<div class="table">
<table border="1" cellspacing="0" style="width: 100%;"><tbody>
<tr><th>ListID</th><th>Subscribers</th><th>Date</th></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>1</td><td>1000</td><td>2017-10-01</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>2</td><td>2000</td><td>2017-10-01</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>3</td><td>3000</td><td>2017-10-01</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>4</td><td>4000</td><td>2017-10-01</td></tr>
<tr align="right"><td>1</td><td>1000</td><td>2017-10-02</td></tr>
<tr align="right"><td>2</td><td>2003</td><td>2017-10-02</td></tr>
<tr align="right"><td>3</td><td>3001</td><td>2017-10-02</td></tr>
<tr align="right"><td>4</td><td>3998</td><td>2017-10-02</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>1</td><td>1000</td><td>2017-10-03</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>2</td><td>2003</td><td>2017-10-03</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>3</td><td>3001</td><td>2017-10-03</td></tr>
<tr align="right" style="background-color: #e9f0db; color: black;"><td>4</td><td>4002</td><td>2017-10-03</td></tr>
</tbody></table></div>
<br />
And this can then be used when reporting over date ranges from your front end reporting application.<br />
<br />
<div class="graf graf--p" name="099f">
This example is just one of many use-cases where Matillion has helped Campaign Monitor fulfill its ETL requirements in order to provide reporting on large datasets. In particular, event based data has helped significantly improve SLAs since the frequency of aggregating data can be done within minutes compared to that of the traditional relational data ETLs that process large queries over many hours and provide much lower refresh rates for report data.</div>
</div>
</div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com1tag:blogger.com,1999:blog-6898784454956902947.post-15273873949189046212017-09-13T02:52:00.001+00:002017-09-13T02:52:40.777+00:00Gotchas with Altering Table Schemas in Redshift<div dir="ltr" style="text-align: left;" trbidi="on">
As I learned recently the hard way, simply altering tables in Redshift to perform a simple operation of adding columns doesn't behave the same way in a data warehouse as it does in regular relational databases.<br />
<br />
Running the following SQL statement in Redshift actually has a lot more issues than you would expect:<br />
<br />
<pre><span style="color: blue; font-family: "consolas"; font-size: 10.5pt; line-height: 107%;">ALTER</span><span style="font-family: "consolas"; font-size: 10.5pt; line-height: 107%;"> <span style="color: blue;">TABLE</span> rs_table <span style="color: blue;">ADD</span> <span style="color: blue;">COLUMN</span> new_column<span style="color: grey;">;</span></span>
</pre>
<br />
Despite the simple nature of this statement, many issues begin to emerge.<br />
<br />
One of these is table locking. If an application is connecting to it via JDBC, connections would end up being queued because one Redshift cluster only allows 500 concurrent connections as per their <a href="http://docs.aws.amazon.com/redshift/latest/mgmt/amazon-redshift-limits.html">online documentation</a>.<br />
<br />
Moreover, it was noticed that there were a number of duplicate records when inserts occurred on a matching key instead of updating the existing record or deleting the old record. This was most likely due to the fact that it was in the process of being altered during an insert and the key match could not be found.<br />
<div>
<br /></div>
<div>
This meant that an alternative approach was needed in order to prevent the locking and concurrency.</div>
<div>
The following steps were tried and tested, and proved to work well:</div>
<div>
<ul style="text-align: left;">
<li>Create the new table with the new schema suffixed with _new</li>
<li>Copy data from the old table to the new table</li>
<li>Rename old table suffixed with _old</li>
<li>Rename new table removing _new suffix</li>
</ul>
</div>
<div>
This is done in SQL as follows:</div>
<div>
<br /></div>
<div>
<pre><div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="color: green; font-family: "consolas"; font-size: 10.5pt;">-- Create new table with new schema
</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="color: blue; font-family: "consolas"; font-size: 10.5pt;">CREATE</span><span style="font-family: "consolas"; font-size: 10.5pt;"> <span style="color: blue;">TABLE</span> rs_table_new
</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="font-family: "consolas"; font-size: 10.5pt;">(
</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="font-family: "consolas"; font-size: 10.5pt;"> id <span style="color: blue;">numeric</span><span style="color: grey;">(</span>10<span style="color: grey;">),
</span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="font-family: "consolas"; font-size: 10.5pt;"> name <span style="color: blue;">varchar</span><span style="color: grey;">(</span>250<span style="color: grey;">),
</span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="font-family: "consolas"; font-size: 10.5pt;"> <span style="color: blue;">date</span> <span style="color: blue;">timestamp</span><span style="color: grey;">,
</span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="font-family: "consolas"; font-size: 10.5pt;"> new_column <span style="color: blue;">numeric</span><span style="color: grey;">(</span>10<span style="color: grey;">)
</span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="font-family: "consolas"; font-size: 10.5pt;">)</span><span style="font-family: "consolas"; font-size: 10.5pt;">
SORTKEY<span style="color: grey;">(</span><span style="color: blue;">dbid</span><span style="color: grey;">,</span> <span style="color: blue;">date</span><span style="color: grey;">,</span> clientid<span style="color: grey;">);</span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
</div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="color: green; font-family: "consolas"; font-size: 10.5pt;">-- Copy data from old table to new
</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="color: blue; font-family: "consolas"; font-size: 10.5pt;">INSERT</span><span style="font-family: "consolas"; font-size: 10.5pt;"> <span style="color: blue;">INTO</span> rs_table_new
</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="color: blue; font-family: "consolas"; font-size: 10.5pt;">SELECT
</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="font-family: "consolas"; font-size: 10.5pt;"> id<span style="color: grey;">,
</span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="font-family: "consolas"; font-size: 10.5pt;"> name<span style="color: grey;">,
</span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="font-family: "consolas"; font-size: 10.5pt;"> <span style="color: blue;">date</span><span style="color: grey;">,
</span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="font-family: "consolas"; font-size: 10.5pt;"> <span style="color: grey;">NULL
</span></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="color: blue; font-family: "consolas"; font-size: 10.5pt;">FROM </span><span style="font-family: "consolas"; font-size: 10.5pt;">rs_table<span style="color: grey;">;</span><o:p></o:p></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
</div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="color: green; font-family: "consolas"; font-size: 10.5pt;">-- Rename table
</span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="color: blue; font-family: "consolas"; font-size: 10.5pt;">ALTER</span><span style="font-family: "consolas"; font-size: 10.5pt;"> <span style="color: blue;">TABLE</span> rs_table RENAME <span style="color: blue;">TO </span>rs_table_old<span style="color: grey;">;</span><o:p></o:p></span></div>
<div class="MsoNormal" style="line-height: normal; margin-bottom: .0001pt; margin-bottom: 0cm; mso-layout-grid-align: none; text-autospace: none;">
<span style="color: blue; font-family: "consolas"; font-size: 10.5pt;">ALTER</span><span style="font-family: "consolas"; font-size: 10.5pt;"> <span style="color: blue;">TABLE</span> rs_table_new RENAME <span style="color: blue;">TO </span>rs_table<span style="color: grey;">;</span></span></div>
</pre>
</div>
<div>
<br /></div>
<div>
Amazingly, the total execution time for a table with over 10 million records completed in under 2 minutes as opposed to the initial 2 hours.</div>
<div>
<br /></div>
</div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com1tag:blogger.com,1999:blog-6898784454956902947.post-48405815007502890902016-06-13T01:09:00.000+00:002022-02-03T05:22:42.022+00:00Don't Always Believe Your Data<div dir="ltr" style="text-align: left;" trbidi="on">
As erroneous as that statement might seem, not trusting your data might be the most prudent thing you do. In this new age of data-driven development where data has become a precious commodity, the need to unlock valuable information that businesses hold about customer interactions has become a crucial part of business success, and hence the investment for better ways to store and extract data has evolved over the last few years more than it ever had over the previous two decades.<br />
<br />
The options are endless now with experts recommending vast arrays of strategies to tackle your storage and analysis techniques. But it is more than just the sum of all knowledge about these alternatives that makes you a Data Scientist. The reason we store data in the first place is to provide information that can remove ambiguity in our decision making process. That requires a broader outlook towards investigating industry trends and changes in technology on top of our expertise on how well we persist and describe our business data.<br />
<br />
There was a time when relational data stores ruled the earth and with them came the ability to analyse large amounts of data using OLAP cubes. But as the need grew for a more efficient Extract Transform Load (ETL) process the limitations became obvious. Analyzing records that were aggregated nightly because there was “too much data” to extract no longer became a legitimate excuse. Businesses want it now, and they want a lot of it.<br />
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisQ0Ue_FSidSblQGIvWGVGCMpmCjV85oqITWMIEX6NMuWqO1Z6cjzkoiTDuZvPxX9Evea_UiHSkF2Pi22COHs_PZknfY27oi0HzmHCPD45bOIBC6WzdES-EJzhDHEKW1zFmK_CMLjw_hKK/s1600/nosql-databases.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="142" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisQ0Ue_FSidSblQGIvWGVGCMpmCjV85oqITWMIEX6NMuWqO1Z6cjzkoiTDuZvPxX9Evea_UiHSkF2Pi22COHs_PZknfY27oi0HzmHCPD45bOIBC6WzdES-EJzhDHEKW1zFmK_CMLjw_hKK/s200/nosql-databases.png" width="200" /></a></div>
<div class="graf graf--p" name="bc36">
With that, storage capabilities evolved to scale horizontally with the map-reduce patterns of <em class="markup--em markup--p-em">document stores</em> like Mongo, <em class="markup--em markup--p-em">key-value stores</em> like Riak, <em class="markup--em markup--p-em">graph stores</em> like Neo4j, and <em class="markup--em markup--p-em">columnar stores</em> like Cassandra. Add into the mix the likes of Big Data storage such as Hadoop, and Event Streams for real time processing like kafka and Amazon Kinesis. Although competition is always a welcome dynamic in any industry, with every contender vying for feature capabilities over their competitors, the choice becomes difficult.</div>
<br />
Storage is just one part of the puzzle however. Next would come analysis and forecast. Once a business makes sense of the data it holds with a myriad of techniques for analysis, predicting change comes packaged for us with Machine Learning. Apache Spark jumped on this wagon early commercially with its MLlib offering and Google has also been a popular choice with TensorFlow. There is a degree of expertise required in handling these tools including understanding the type of learning you want your machine to perform, whether its <i>supervised learning</i> using labeled datasets (for fraud detection and recommendations), <i>unsupervised and semi-supervised learning</i> with unlabeled datasets (for image and voice recognition), or <i>reinforcement learning</i> (for artificial intelligence).<br />
<br />
There is more to it than knowing your alternatives though. Your data can only tell you as much information as your business keeps about itself, and so Data Science is not just about <b><i>how</i></b> to unlock your data with the available tools, its about causation. It's about <b><i>why</i></b>.<br />
<br />
Identifying peaks or troughs in a graph might help you determine with a guarantee that your tools can help you see patterns or changes, but not the reasons why these patterns and changes exist. And thus starts the real investigation.<br />
<br />
This was a lesson that Andrea Burbank from Pinterest learnt as she explained at the <a href="http://yowconference.com.au/">YOW! Conference</a> held in Sydney late 2016, and she was kind enough to share her challenges in trying to successfully formulate behavioral trends from Pinterest's massive data storage. The results of her findings were very interesting.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidnnNwCofXAnEP_e5lWZAhaRczGOxqbU93yUeRXSxrdLDohCeTEmKTTkr2V5MytGwwlb_i1v14tEdufxGVghtY7tKmLMVAtnGWIYeaONIIGGZFO6tCgAPcNUoe5Apj7rHN7RDmDttxRiET/s1600/ios.user.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="220" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidnnNwCofXAnEP_e5lWZAhaRczGOxqbU93yUeRXSxrdLDohCeTEmKTTkr2V5MytGwwlb_i1v14tEdufxGVghtY7tKmLMVAtnGWIYeaONIIGGZFO6tCgAPcNUoe5Apj7rHN7RDmDttxRiET/s320/ios.user.png" width="320" /></a>By measuring "daily active users", Pinterest were able to determine which unique customers engaged on their site. Using kafka, they were able to stream logs of response data that told the user's story, but that was just basic counting.<br />
<br />
In late October of 2013, they found a sudden step change in growth rate specifically with iPhone users, which seemed likely to be the result of Pinterest being featured in the App Store. What they found though was that it was merely a coincidence because not long before a new feature was introduced to iOS 7 called "Background App Refresh" to prevent apps hanging in suspended animation.<br />
<br />
Burbank realized that the statistics they generated weren't telling the full story and after digging further found that one of their endpoints, particularly the home feed endpoint, was giving unnatural background hits even when the app wasn't appearing in the multitask tray, and that obviously had no correlation to daily active users.<br />
<br />
This lead to what Burbank defined as the "Spectrum of Certainty" where <i>correlation</i> is found at one end of the spectrum while <i>causation</i> is found at the other. Most of the time Data Scientists believe they are closer to the causation end of the spectrum, when in actual fact they sit very close to the correlation side.<br />
<br />
The goal is to ensure data analysis moves towards causal inference, in effect requiring much more investigative techniques about events that occur outside the business domain which may have a strong impact on how data is represented within the business itself. Only then can you truly believe your data to a certain degree.<br />
<br />
Reference: <a href="http://yowconference.com.au/slides/yow2016/Burbank-DataScienceAsSoftware.pdf">Data science as software, Andrea Burbank, Pinterest, YOW! Conference 2016 Sydney</a></div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0tag:blogger.com,1999:blog-6898784454956902947.post-30483166615976655452015-12-18T01:39:00.003+00:002020-02-10T09:38:31.611+00:00INNER JOIN vs WHERE EXISTS Sub-Queries in MSSQL<div dir="ltr" style="text-align: left;" trbidi="on">
Currently working at WiseTech, I've learn't a great deal about the pros and cons of monolithic applications, and more so about why microservices have been a massive talking point in the industry today. There are many specialists in the technology stack here and recently I got into a discussion about SQL performance tuning.<br />
<br />
For a long time I always thought there was a standard golden rule when it came to arguing against sub-queries, and it was almost as sacrilege as cursors, however there have been a few circumstances where the difference may actually vary very little.<br />
<br />
Here is a very simple statement that contains one INNER JOIN:<br />
<br />
<pre><div class="MsoNormal">
<span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">SELECT</span><span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: grey;">*</span><o:p></o:p></span></div>
<div class="MsoNormal">
<span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">FROM</span><span style="font-family: "consolas"; font-size: 9.5pt;"> StmNote<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="background-color: yellow; font-family: "consolas"; font-size: 9.5pt;">INNER</span><span style="font-family: "consolas"; font-size: 9.5pt;"><span style="background-color: yellow;"> <span style="color: grey;">JOIN</span> JobDeclaration <span style="color: blue;">ON</span>
JE_PK <span style="color: grey;">=</span> ST_ParentID</span><o:p></o:p></span></div>
<div class="MsoNormal">
<span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">WHERE</span><span style="font-family: "consolas"; font-size: 9.5pt;"> ST_Description <span style="color: grey;">=</span> <span style="color: red;">'Justification Note'</span><o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: grey;">AND</span>
ST_Table <span style="color: grey;"><></span> <span style="color: red;">'JobDeclaration'</span><o:p></o:p></span><span style="font-family: "consolas"; font-size: 12.6667px;">;</span></div>
</pre>
<br />
And below is an identical query that returns the same results, only with the difference being that it contains a WHERE EXISTS sub-query clause instead of an INNER JOIN:<br />
<br />
<pre><div class="MsoNormal">
<span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">SELECT</span><span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: grey;">*</span><o:p></o:p></span></div>
<div class="MsoNormal">
<span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">FROM</span><span style="font-family: "consolas"; font-size: 9.5pt;"> StmNote<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">WHERE</span><span style="font-family: "consolas"; font-size: 9.5pt;"> ST_Description <span style="color: grey;">=</span> <span style="color: red;">'Justification Note'</span><o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: grey;">AND</span>
ST_Table <span style="color: grey;"><></span> <span style="color: red;">'JobDeclaration'</span><o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: grey;">AND</span> <span style="color: grey;">EXISTS</span><span style="color: blue;"> </span><span style="color: grey;">(</span><span style="background-color: yellow;"><span style="color: blue;">SELECT</span> 1 <span style="color: blue;">FROM</span> JobDeclaration <span style="color: blue;">WHERE</span>
JE_PK <span style="color: grey;">=</span> ST_ParentID</span><span style="color: grey;">);</span></span></div>
</pre>
<br />
The below execution plan proves there is absolutely no difference in the decision path that the query optimizer makes.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNUdO1L6NwgbtaNkHieICMzRJt2rP9rwUlDmi1dZlgZFHZmjaxE8ade-7GvrYOv0I_rcXed_IfA4aMh0tOumX0RFINSf0G2g2rveRS4Ij7UJKv3z8LjmNgtseN3AcANLpAtx5drWeSSaGx/s1600/inner.vs.where.exists.simple.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="260" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNUdO1L6NwgbtaNkHieICMzRJt2rP9rwUlDmi1dZlgZFHZmjaxE8ade-7GvrYOv0I_rcXed_IfA4aMh0tOumX0RFINSf0G2g2rveRS4Ij7UJKv3z8LjmNgtseN3AcANLpAtx5drWeSSaGx/s640/inner.vs.where.exists.simple.png" width="640" /></a></div>
<br />
This could easily make one argue that the difference in both statements would simply come down to a matter of preference.<br />
<br />
However as we know the query optimizer behaves very differently when considering the number of rows involved in an index-seek/table-scan and the statistics it has to work with in determining and efficient execution path.<br />
<br />
So what if there were other joins or larger result sets which could ultimately mean more variations in the execution path for query optimizer?<br />
<br />
Below is a more complex scenario involving multiple joins that in the first case uses the following query:<br />
<br />
<pre><div class="MsoNormal">
<span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">SELECT</span><span style="font-family: "consolas"; font-size: 9.5pt;"> bna<span style="color: grey;">.*</span> <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">FROM</span><span style="font-family: "consolas"; font-size: 9.5pt;"> BmncnAttachment bna<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;">INNER</span><span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: grey;">JOIN</span> </span><span style="font-family: "consolas"; font-size: 12.6667px;">Bmncn</span><span style="font-family: "consolas"; font-size: 9.5pt;">Shape bns<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: blue;">ON</span> bns<span style="color: grey;">.</span>BNS_PK <span style="color: grey;">=</span> bna<span style="color: grey;">.</span>BNA_BNS_FromShape <span style="color: grey;">OR</span>
bns<span style="color: grey;">.</span>BNS_PK <span style="color: grey;">=</span>
bna<span style="color: grey;">.</span>BNA_BNS_ToShape <span style="color: grey;">OR</span>
bns<span style="color: grey;">.</span>BNS_PK <span style="color: grey;">=</span>
bna<span style="color: grey;">.</span>BNA_BNS_Owner<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="background-color: yellow; font-family: "consolas"; font-size: 9.5pt;">INNER</span><span style="font-family: "consolas"; font-size: 9.5pt;"><span style="background-color: yellow;"> <span style="color: grey;">JOIN</span> ProcessHeader fh</span><o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="background-color: yellow;"><span style="color: blue;">ON</span> bns<span style="color: grey;">.</span>BNS_FH_ProcessHeader <span style="color: grey;">=</span>
fh<span style="color: grey;">.</span>FH_PK</span><o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;">INNER</span><span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: grey;">JOIN</span> WorkItem wki<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: blue;">ON</span> wki<span style="color: grey;">.</span>WKI_PK <span style="color: grey;">IS</span> <span style="color: grey;">NOT</span> <span style="color: grey;">NULL</span><o:p></o:p></span></div>
<div class="MsoNormal">
<span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">WHERE</span><span style="font-family: "consolas"; font-size: 9.5pt;"> wki<span style="color: grey;">.</span>WKI_PK <span style="color: grey;">=</span> fh<span style="color: grey;">.</span>FH_ParentId<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: grey;">AND</span> wki<span style="color: grey;">.</span>WKI_Summary <span style="color: grey;">LIKE</span> <span style="color: red;">'Issue %'</span><o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: grey;">AND</span> wki<span style="color: grey;">.</span>WKI_Status <span style="color: grey;">IN</span><span style="color: blue;"> </span><span style="color: grey;">(</span><span style="color: red;">'ASN'</span><span style="color: grey;">,</span> <span style="color: red;">'OPN'</span><span style="color: grey;">,</span> <span style="color: red;">''</span><span style="color: grey;">)</span><o:p></o:p></span><span style="font-family: "consolas"; font-size: 12.6667px;">;</span></div>
</pre>
<br />
While the second case uses the following query with a WHERE EXISTS sub-query clause:<br />
<br />
<pre><div class="MsoNormal">
<span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">SELECT</span><span style="font-family: "consolas"; font-size: 9.5pt;"> bna<span style="color: grey;">.*</span> <o:p></o:p></span></div>
<div class="MsoNormal">
<span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">FROM</span><span style="font-family: "consolas"; font-size: 9.5pt;"> </span><span style="font-family: "consolas"; font-size: 12.6667px;">Bmncn</span><span style="font-family: "consolas"; font-size: 9.5pt;">Attachment bna<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;">INNER</span><span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: grey;">JOIN</span> </span><span style="font-family: "consolas"; font-size: 12.6667px;">Bmncn</span><span style="font-family: "consolas"; font-size: 9.5pt;">Shape bns<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: blue;">ON</span> bns<span style="color: grey;">.</span>BNS_PK <span style="color: grey;">=</span> bna<span style="color: grey;">.</span>BNA_BNS_FromShape <span style="color: grey;">OR</span>
bns<span style="color: grey;">.</span>BNS_PK <span style="color: grey;">=</span>
bna<span style="color: grey;">.</span>BNA_BNS_ToShape <span style="color: grey;">OR</span>
bns<span style="color: grey;">.</span>BNS_PK <span style="color: grey;">=</span>
bna<span style="color: grey;">.</span>BNA_BNS_Owner<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;">INNER</span><span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: grey;">JOIN</span> WorkItem wki<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: blue;">ON</span> wki<span style="color: grey;">.</span>WKI_PK <span style="color: grey;">IS</span> <span style="color: grey;">NOT</span> <span style="color: grey;">NULL</span><o:p></o:p></span></div>
<div class="MsoNormal">
<span style="color: blue; font-family: "consolas"; font-size: 9.5pt;">WHERE</span><span style="font-family: "consolas"; font-size: 9.5pt;"> wki<span style="color: grey;">.</span>WKI_Summary <span style="color: grey;">LIKE</span> <span style="color: red;">'Issue %'</span><o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: grey;">AND</span> wki<span style="color: grey;">.</span>WKI_Status <span style="color: grey;">IN</span><span style="color: blue;"> </span><span style="color: grey;">(</span><span style="color: red;">'ASN'</span><span style="color: grey;">,</span> <span style="color: red;">'OPN'</span><span style="color: grey;">,</span> <span style="color: red;">''</span><span style="color: grey;">)</span><o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: grey;">AND</span> <span style="color: grey;">EXISTS</span><span style="color: blue;"> </span><span style="color: grey;">(</span><span style="background-color: yellow;"><span style="color: blue;">SELECT</span> 1 <span style="color: blue;">FROM</span> ProcessHeader fh <span style="color: blue;">WHERE</span>
bns<span style="color: grey;">.</span>BNS_FH_ProcessHeader <span style="color: grey;">=</span> fh<span style="color: grey;">.</span>FH_PK <span style="color: grey;">AND</span> wki<span style="color: grey;">.</span>WKI_PK <span style="color: grey;">=</span> fh<span style="color: grey;">.</span>FH_ParentId</span><span style="color: grey;">)</span><o:p></o:p></span><span style="font-family: "consolas"; font-size: 12.6667px;">;</span></div>
</pre>
<br />
And here is the varying execution plans for both:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipMnc4ajFkWxkFWEPPFr8XdB_lPM1kxCUaJFXLDa5AwbL1GT0c5CDYDZZZFHjY2d7DVlrPXjmyTtviUlQCKFuY5GOCRqczQLBrk641enQbETUGQDTW4PL8G9d7YbZnE-bJXoThd7nrpkCA/s1600/inner.vs.where.exists.complex.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="414" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipMnc4ajFkWxkFWEPPFr8XdB_lPM1kxCUaJFXLDa5AwbL1GT0c5CDYDZZZFHjY2d7DVlrPXjmyTtviUlQCKFuY5GOCRqczQLBrk641enQbETUGQDTW4PL8G9d7YbZnE-bJXoThd7nrpkCA/s640/inner.vs.where.exists.complex.png" width="640" /></a></div>
<br />
<br />
<br />
It seems they differ significantly on the final Hash Matching where it is probing the hash key fh.FH_ParentId and taking 62% or the total time longer than the standard inner join. The is mainly because you lose parallelism when using the sub-query.<br />
<br />
So basically to sum it all up, even though the first statement may show identical results, performance issues would exists if there were multiple joins, because the optimizer could choose the sub-query over another join which would otherwise have been more efficient.<br />
<div>
<br /></div>
<div>
Conversely however, performance could be effected for joins that return duplicate rows and that may be another variable to consider when making the decision between WHERE EXIST sub-query clauses and JOINS, and is definitely something to keep in mind when making the final decision on what type of statement to use.</div>
<div>
<br /></div>
</div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0tag:blogger.com,1999:blog-6898784454956902947.post-16612711256117681412015-09-16T05:27:00.000+00:002020-02-10T09:40:09.862+00:00Is Server-Side Javascript the Flavor of the Month?<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpcqhA1xJrortI0QGbmwAowjFPtDhpLt7pHUhLfJ_PgftP1bZUzESql6X7PvQgdggB5WvDA7zqi8A75IDzjOirWuugjedd_naXD4tFUuu-9dzB1INS_LR90Ok4dQuRyvT7PZ4vd85cRngQ/s1600/tools.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="275" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpcqhA1xJrortI0QGbmwAowjFPtDhpLt7pHUhLfJ_PgftP1bZUzESql6X7PvQgdggB5WvDA7zqi8A75IDzjOirWuugjedd_naXD4tFUuu-9dzB1INS_LR90Ok4dQuRyvT7PZ4vd85cRngQ/s320/tools.png" width="320" /></a></div>
As many experienced developers would tell you, <i>"choose the right tool for the job"</i>. While many may have strong bias towards one programming style over another, it is always good to encounter those who are willing to present an argument objectively.<br />
<br />
These days there are a lot of advocates for Node.js, and they all begin their marketing campaigns with the same opening statements, <i>"I know Javascript is crap but..."</i><br />
<br />
And therein lies the dilemma. Its not an easy task having to justify a scripting language as the solution for web application development in an era of staleness within the open-source community. Lets face it, php has one of the largest online communities on the planet, particularly with CMS solutions like WordPress and Drupal, however the language hasn't evolved or adapted to an ever-changing environment. Ruby as a framework doesn't scale well and Python has a very poor reputation for performance.<br />
<br />
Node.js seems like a breath of fresh air.<br />
<br />
In all honesty, when Node.js came out a few years back with this "new" concept of server-side Javascript, I questioned why we were getting back into the world of Classic ASP which was prevalent in the late 1990s. Back then, we had the power of inline server-side code with your choice of either VBScript or JScript, so the concept wasn't "new" unless you think relatively in terms of the 20 years that this idea has been around.<br />
<br />
It was only a little later that I realised it had nothing to do with "Javascript being server-side", but rather that Node.js was able to tame the beast of a scripting language, and create a discipline using successfully proven concepts such as modularisation and encapsulation.<br />
<br />
These concepts are very familiar in the object-oriented world and have been around for decades. So why create another server-side framework using Javascript? The easiest answer would most likely be because it is cross-platform and has been jointly adopted and standardized by Microsoft, Sun Microsystems and Mozilla.<br />
<br />
Javascript has been superior in web development and, as Scott Hanselman put it, <i><a href="http://www.hanselman.com/blog/JavaScriptIsWebAssemblyLanguageAndThatsOK.aspx">"is the assembly language of the web"</a></i>. It is uncontested as a front-end solution for interactive browser-based development, which is why the adoption of such frameworks like angular.js have been extremely popular, with react.js fiercely becoming a dominant framework too. Bundlers like require.js and webpack are becoming household names, and templating tools such as handlebars.js and jade have shown that Javascript has the power to dominate an ever-growing community. Together with the power of HTML5 and CSS3, it has killed off both Flash and Silverlight in one single stoke.<br />
<br />
But lets be clear, we are talking about front-end development here, not server-side development.<br />
<br />
While searching the online community to find any significant advantages of using Javascript for back-end development using Node.js, there have only really been two common themes:<br />
<ul style="text-align: left;">
<li><i>"Front-end developers can work on the full stack of an application using one language"</i><br />This argument is great, as we have seen with the emergence of the MEAN stack. But the separation of front-end and back-end development has never been one of language barriers. Rather the difference is based on a mind-set. Lets face it, you won't see front-end developers picking up tasks related to data-access or API security. The real question is whether or not a functional scripting language is appropriate for server-side application programming.
<br /><br />
</li>
<li><i>"You can start a web application in literally 5 minutes!"</i><br />While that might be great for those who want a pet project to build "hello world" applications, professional programmers will find obvious challenges once complexity is inevitably added. The headaches around peer dependencies is just one, however the topic of discussion that has been the most entertaining is: to bundle or not to bundle. An issue so primitive it is almost comical to consider that other frameworks solved compilation concerns with output libraries more than 15 years ago. It is a non-issue now and yet somehow it seemed like we have gone back to re-inventing the wheel all over again with Node.js. Those who work on other frameworks with rich environments that provide debugging, diagnostics and compilation already had the wheels they needed to concentrate on real programming to provide the vehicles for those wheels.</li>
</ul>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEib8qR-YZaOegkSbpL9zgHoPpBjQTErqyf9Vgq3z3RUCfmbrYWf4Qjhk_XHzKAz_95c2q6hJ2QC_FlOov6kOEERm5WE9WHKW7L_ZRgYOHwjma8_rU2ZCTbeR47lvSafvRKbeslEHyO17SXI/s1600/zz84iru523uz.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" data-original-height="1600" data-original-width="472" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEib8qR-YZaOegkSbpL9zgHoPpBjQTErqyf9Vgq3z3RUCfmbrYWf4Qjhk_XHzKAz_95c2q6hJ2QC_FlOov6kOEERm5WE9WHKW7L_ZRgYOHwjma8_rU2ZCTbeR47lvSafvRKbeslEHyO17SXI/s320/zz84iru523uz.png" width="94" /></a>Stability in the community is one point Node.js cannot boast about either, especially when recalling how easy it was to <a href="http://www.infoworld.com/article/2854642/javascript/nodejs-gains-a-new-fork-much-to-joyents-dismay.html">fork the code-base</a> and create an entirely new community with io.js simply because they were "dissatisfied" by the Advisory Board. They have recently reunited, but how long will it be before another subgroup becomes fickle and creates another community of their own? Angular.js is already <a href="http://ilikekillnerds.com/2015/01/aurelia-vs-angularjs-round-one-fight/">facing the same problems</a>, which is why aurelia.io has come out as a potential competitor. A unified community generally gives confidence in a framework's future.<br />
<br />
But lets not get into politics.<br />
<br />
Without getting too deep into the "one thread per request" model limitation, or the limitation of only two basic data structures in comparison to other frameworks that provide hash-maps, dictionaries, enumerables, sets, vectors etc, there is one more concerning factor.<br />
<br />
If we take a deeper look into programming principals, then we can start comparing Node.js with enterprise frameworks such as Java and .NET and their object-oriented programming edicts.<br />
<br />
Software engineering principals like OOP adopt concepts like abstraction and polymorphism, with mathematical concepts like covariance and contravariance.<br />
They define the difference between software engineering and simple web development because it is the philosophy that engineers and science professors like <a href="https://en.wikipedia.org/wiki/Alan_Kay">Alan Kay</a> prescribed for application programming in order to help mimic the real world.<br />
<br />
The universe is defined with objects that interact with other objects in one way or another by performing a specific function. Each object has its own attributes, and has general similarities with other existing objects which can be abstracted. Since the universe is infinitely scalable, the very concept of OOP over functional and procedural programming can determine the best fit for scalability in complex web applications, particularly in the space that exists for implementing business functions.<br />
<br />
One may argue that even between different object-oriented programming languages, the health of the community is equally important. Unfortunately the Java community falls short, and nothing could have been clearer when Java8 came out with the "new" concept of lambda expressions, especially when .NET 3.0 introduced it 5 years prior.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtk_pCD6tuERyvfW9GSTW6Rs2FXioE1OdjlChHnLtIJQsx4XrIA5ak-4fu3KAeFD_sQRwtGV42fgiNrZmIGYQ0AunBtfrxHNDyJ8q68a6Muj3TOk5GI5PnblqNA4DB8gFPOPp_-JK49tSa/s1600/languages.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="204" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtk_pCD6tuERyvfW9GSTW6Rs2FXioE1OdjlChHnLtIJQsx4XrIA5ak-4fu3KAeFD_sQRwtGV42fgiNrZmIGYQ0AunBtfrxHNDyJ8q68a6Muj3TOk5GI5PnblqNA4DB8gFPOPp_-JK49tSa/s320/languages.jpg" width="320" /></a>But if "new" is any measure for Node.js being the better option today, is the mere reasoning for a cool new hip way of doing things good enough? What happens when Node.js is 10 years old? Will Node.js outlive its hipster status? And then there are those who are already talking about Golang being the better option for performance and scalability, which definitely raises alarm bells for its longevity.<br />
<br />
If the fact that Node.js is lightweight and uses a dynamically typed scripting language has anything to do with it, then a simpler question arises: What would stop us from building a web application using say PowerShell scripts? Why not? Its possible to write primitive code in PowerShell to open an HTTP port with binding through IIS to stream strings of HTML! Given enough time someone might write a wrapper and call it express.ps1 to make it easier!<br />
<br />
But that would be silly, wouldn't it...<br />
<br />
Going back to the initial statement to <i>"choose the right tool for the job" </i>might help us maintain focus in an ever-changing web space.<br />
<br />
Being language or framework agnostic is key to determining the right tools and it could help us realise the natural strengths of the technologies at our disposal. Javascript is excellent for front-end development, PowerShell is suitable for systems programming, and frameworks like .NET are great for scalable server-side application programming.<br />
<br /></div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0tag:blogger.com,1999:blog-6898784454956902947.post-45677836397267947962015-07-14T05:40:00.000+00:002015-09-22T00:27:45.053+00:00Resetting CSS3 Transitions and Animations<div dir="ltr" style="text-align: left;" trbidi="on">
Everyone gets excited whenever they get the opportunity to work with CSS3 and particularly the new transition and animation features. They work smoothly compared to Javascript animations, and the reason they do is because these visual interactions are managed by the browser itself and how it decides to interpret the movement of elements that it has rendered from markup. This is significantly more efficient than how Javascript tries manipulating DOM elements functionally by <i>altering </i>the rendered elements frame by frame.<br />
<br />
With every great thing however, there are always shortfalls. Lets look specifically at animations and their keyframes.<br />
<br />
<pre><div class="MsoNormal" style="font-family: monospace; margin-bottom: 0.0001pt; white-space: pre;">
<span style="background: white; color: maroon; font-family: Consolas; font-size: 9.5pt;">.notify</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> {</span></div>
<div class="MsoNormal" style="font-family: monospace; margin-bottom: 0.0001pt; white-space: pre;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: red; font-family: Consolas; font-size: 9.5pt;">animation</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">: </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt;">notify</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt;">5s</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">;</span></div>
<div class="MsoNormal" style="font-family: monospace; margin-bottom: 0.0001pt; white-space: pre;">
<span style="background-color: white; font-family: Consolas; font-size: 9.5pt;">}</span></div>
<div style="font-family: monospace; white-space: pre;">
</div>
<div class="MsoNormal" style="font-family: monospace; margin-bottom: 0.0001pt; white-space: pre;">
<span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt;">@keyframes</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt;">notify</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div style="font-family: monospace; white-space: pre;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: maroon; font-family: Consolas; font-size: 9.5pt;">from</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> { </span><span style="background: white; color: red; font-family: Consolas; font-size: 9.5pt;">height</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">: </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt;">0</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">; }</span></div>
<div style="font-family: monospace; white-space: pre;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: maroon; font-family: Consolas; font-size: 9.5pt;">20%</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> { </span><span style="background: white; color: red; font-family: Consolas; font-size: 9.5pt;">height</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">: </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt;">60px</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">; }</span></div>
<div style="font-family: monospace; white-space: pre;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: maroon; font-family: Consolas; font-size: 9.5pt;">80%</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> { </span><span style="background: white; color: red; font-family: Consolas; font-size: 9.5pt;">height</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">: </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt;">60px</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">; }</span></div>
<div style="font-family: monospace; white-space: pre;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: maroon; font-family: Consolas; font-size: 9.5pt;">to</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> { </span><span style="background: white; color: red; font-family: Consolas; font-size: 9.5pt;">height</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">: </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt;">0</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">; }</span></div>
<div style="font-family: monospace; white-space: pre;">
<span style="background-color: white; font-family: Consolas; font-size: 9.5pt;">}</span></div>
</pre>
<br />
This is a very simple implementation of a keyframe animation to slide an element down for notifications. All that is needed is a div element that initially doesn't have a "notify" class.<br />
<br />
<pre><div class="MsoNormal" style="font-family: monospace; margin-bottom: 0.0001pt; white-space: pre;">
<span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt;"><</span><span style="background: white; color: maroon; font-family: Consolas; font-size: 9.5pt;">div</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: red; font-family: Consolas; font-size: 9.5pt;">id</span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt;">="tip"</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: red; font-family: Consolas; font-size: 9.5pt;">class</span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt;">="tip"></span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">message</span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt;"></</span><span style="background: white; color: maroon; font-family: Consolas; font-size: 9.5pt;">div</span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt;">></span></div>
</pre>
<br />
With a little Javascript we can add the "notify" class to the element so that the keyframe can be triggered. We can do this with JQuery.<br />
<br />
<pre><div class="MsoNormal">
<span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">$(</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-highlight: white;">".tip"</span><span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">).addClass(</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-highlight: white;">"notify"</span><span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">);</span><o:p></o:p></div>
</pre>
<br />
There is one problem with this however. Once this keyframe has been executed, it can no longer be executed unless you refresh the page.<br />
<br />
One way which should typically solve this problem would be to remove the class then re-add it again.<br />
<br />
<pre><div class="MsoNormal">
<span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">$(</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-highlight: white;">".tip"</span><span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">).removeClass(</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-highlight: white;">"notify"</span><span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">);</span><br />
<span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">
$(</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-highlight: white;">".tip"</span><span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">).addClass(</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-highlight: white;">"notify"</span><span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">);</span><o:p></o:p></div>
</pre>
<br />
That would be logical, unfortunately it doesn't work!<br />
Amazingly, what is required is to set any property of the element, even if it is to the same value.<br />
<br />
<pre><span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">$(</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-ansi-language: EN-AU; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: white;">".tip"</span><span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">).removeClass(</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-ansi-language: EN-AU; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: white;">"notify"</span><span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">);
</span>
<span style="background: yellow; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">$(</span><span style="background: yellow; color: #a31515; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-ansi-language: EN-AU; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: yellow;">".tip"</span><span style="background: yellow; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">).position().top = $(</span><span style="background: yellow; color: #a31515; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-ansi-language: EN-AU; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: yellow;">".tip"</span><span style="background: yellow; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">).position().top;
</span>
<span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">$(</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-ansi-language: EN-AU; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: white;">".tip"</span><span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">).addClass(</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-ansi-language: EN-AU; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: white;">"notify"</span><span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">);</span>
</pre>
<br />
The reason for this is it helps trigger a "reflow" of the browser so as to listen for the next time the "notify" class has been added to the element, and the browser will do its magic all over again!<br />
<br />
<br /></div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0tag:blogger.com,1999:blog-6898784454956902947.post-89288450322590322922014-11-14T01:27:00.000+00:002022-02-03T05:21:20.633+00:00Regex for Unicode Variations<div dir="ltr" style="text-align: left;" trbidi="on">
Ever had a spammer try to get around spam rules for a subject line like "Account Verification" which would otherwise be picked up by your regular spam? They would usually try something sneaky like "Acc0unt Verification" by replace o's with zeros (0) or have colons in between so your pattern matching wont pick up subject lines like "A:ccount Verification".<br />
<br />
Worse they would even use subtle variations in vowels to substitute an e with a unicode variation like é to make pattern matching extremely difficult "Accøünt Vérificatiøn".<br />
<br />
Regular expression can definitely make your life easier, but unicode would make things more interesting. First thing you will need is the basic pattern matching to catch out the characters in between that usually tries to through out whole word pattern matches:<br />
<br />
<pre><div class="MsoNormal" style="font-family: monospace; margin-bottom: 0.0001pt; white-space: pre;">
<span style="background: white; color: blue; font-family: "consolas"; font-size: 9.5pt;">^(?i).*(a.?c.?c.?o.?u.?n.?t.?[\s]*v.?e.?r.?i.?f.?i.?c.?a.?t.?i.?o.?n)+.*$</span></div>
</pre>
<br />
You will probably want to add some basic variations to the vowels which would catch any spammer trying to replace o's with zeros (0) or i's with ones (1) or a's with at symbols (@).
<br />
<pre><div class="MsoNormal" style="font-family: monospace; margin-bottom: 0.0001pt; white-space: pre;">
<span style="background: white; color: blue; font-family: "consolas"; font-size: 9.5pt;">
</span>
<span style="background: white; color: blue; font-family: "consolas"; font-size: 9.5pt;">^(?i).*([a@].?c.?c.?[o0].?u.?n.?t.?[\s]*v.?[e3].?r.?[il1].?f.?[il1].?c.?[a@].?t.?[il1].?[o0].?n)+.*$</span></div>
</pre>
<br />
And if you really want to be pedantic, I've found a good table listing all the <a href="http://unicode-table.com/en/">unicode characters</a> and have provided them below so you can replace the vowel matching with the following unicode variations.<br />
<br />
<pre class="MsoNormal" style="margin-bottom: 0.0001pt; white-space: pre; background: white; color: blue; font-family: "consolas"; font-size: 9.5pt;">
[a@À-Åà-åĀ-ąǍǎǞ-ǡǺǻȀ-ȃȦȧȺɑΆΑάαаӐ-ӓᗅᶏᶐḀḁẚẠ-ặἈ-ἏÅᾈ-ᾏᾸ-ᾼ₳]<br />
[e3È-Ëè-ëĒ-ěȄ-ȇȨȩɆɇΈΕеѐёҼ-ҿӖӗᴇḔ-ḝẸ-ệⴹἘ-ἝῈΈ]<br />
[il1Ì-Ïì-ïĨ-ıĺļľŀłƖƗǏǐȈ-ȋɨ-ɭΊΙіїӀḬ-ḯḷ-ḽỈ-ịἰ-Ἷὶίῐ-Ί]<br />
[o0Ò-Øð-øŌ-őǑǒǪ-ǭǾǿȌ-ȏȪ-ȱɸɵΌΟθоѲѳӦ-ӫ০੦௦ᴏṌ-ṓỌ-ợὀ-Ὅὸό]<br />
[uµÙ-Üù-üŨ-ųǓ-ǜȔ-ȗɄᴜṲ-ṻỤ-ựὐ-ὗὺύῠ-ΰ]<br />
</pre>
<br />
There are a few common consonants that also get used by spammers to avert the pattern matching:
<br />
<pre class="MsoNormal" style="margin-bottom: 0.0001pt; white-space: pre; background: white; color: blue; font-family: "consolas"; font-size: 9.5pt;">
[nŃ-ŋƝƞǸǹɳΝᶇṄ-ṋἠ-ἧ]<br />
[tŢ-ŧƫ-ƮȚțȾᴛṪ-ṱ]<br />
[y¥ÝýÿŶ-ŸƳƴȲȳɎɏΫγϒ-ϔўҮ-ұӮ-ӳẎẏẙỲ-ỹỾỿὙ-ὟⲨⲩ]<br />
</pre>
The end result of the regex pattern may seem long and complicated, but can definitely prevent many spam emails from slipping through the cracks
</div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0tag:blogger.com,1999:blog-6898784454956902947.post-76170073298224232122014-05-27T03:02:00.002+00:002014-06-10T06:45:57.082+00:00Javascript's Decorator Pattern using Require.js<div dir="ltr" style="text-align: left;" trbidi="on">
While working at Mi9 I stumbled across a very interesting way to use the <a href="http://addyosmani.com/resources/essentialjsdesignpatterns/book/#decoratorpatternjavascript">Javascript Decorator Pattern</a> in order to separate the concerns of the html and the way it populates information using the require.js framework.<br />
<br />
For those of you who are unfamiliar to <a href="http://requirejs.org/">require.js</a>, I have provided an extremely simple example below. It is basically a simple lightweight framework that helps you load javascript files on demand. Each of these files provides a module that can introduce features to your javascript functionality.<br />
<br />
A simple example of require.js is can be shown below when you specify a module in the require construct, which uses dependency injection to pass in the <span style="font-family: Courier New, Courier, monospace;">helper</span> service and its members:<br />
<br />
<pre><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> require([</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">'jquery'</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">, </span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">'</span><span style="background-attachment: initial; background-clip: initial; background-color: yellow; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: #a31515; font-family: Consolas; font-size: 9.5pt;">./helper</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">'</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">], </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">function</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> ($, </span><span style="background-attachment: initial; background-clip: initial; background-color: yellow; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;">helper</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">) {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
</div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">var</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> callSingleStoryService = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">function</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> (url) {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> $.get(url, </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">function</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> (data) {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; font-family: Consolas; font-size: 9.5pt;"><span style="background-color: white;">
</span><span style="background-color: yellow;">helper.evaluateElapsedTime(data)</span></span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">;<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> });<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> };<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
</div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">return</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> decorate: </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">function</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> (data) {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> callSingleStoryService(data);<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> };<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;"> });</span><o:p></o:p></div>
</pre>
<br />
This will locate the helper file located relatively in ./helper, and the actual helper module provides the functionality for <span style="font-family: Courier New, Courier, monospace;">evaluateElapsedTime(data);</span> as shown below:<br />
<br />
<pre><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> define([</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">'jquery'</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">], </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">function</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> ($) {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
</div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">return</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> evaluateElapsedTime: </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">function</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> (element) {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: green; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">//do stuff...</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> };<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;"> });</span><o:p></o:p></div>
</pre>
<br />
This is just a simple example of how to use require.js.<br />
<br />
Now, to apply this to an html element in order to provide a decorator to its inner elements, you can provide the following attributes as follows in your view:<br />
<br />
<pre><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;"><</span><span style="background: white; color: maroon; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">div</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: red; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">class</span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">="module"</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background-color: yellow;"><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: red; font-family: Consolas; font-size: 9.5pt;">data-module</span><span style="background-attachment: initial; background-clip: initial; background-image: initial; background-origin: initial; background-position: initial; background-repeat: initial; background-size: initial; color: blue; font-family: Consolas; font-size: 9.5pt;">="./feedDecorator"</span></span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">></span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;"><</span><span style="background: white; color: maroon; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">div</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: red; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">class</span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">="block"></span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;"><</span><span style="background: white; color: maroon; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">div</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: red; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">class</span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">="grid"></span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> Some content...<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;"></</span><span style="background: white; color: maroon; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">div</span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">></span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;"></</span><span style="background: white; color: maroon; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">div</span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">></span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;"></</span><span style="background: white; color: maroon; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">div</span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">></span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"><o:p></o:p></span></div>
</pre>
<br />
<div class="MsoNormal">
<br /></div>
The above html tells the application to associate the feedDecorator.js file which has a decorate function in it to apply functionality to this specific element with a "module" class name. Your feedDecorator.js file will look like the following:<br />
<br />
<pre><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;">(</span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">function</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> () {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> define([</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">'jquery'</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">, </span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">'./helper'</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">], </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">function</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> ($, helper) {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt;">return</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> {</span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> decorate: </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">function</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> ($target, data) {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> helper.evaluateElapsedTime($target,
data);<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> };<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> });<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">});</span><o:p></o:p></div>
</pre>
<br />
How does it all work? Your main.js (the only js file that you need to initially load) should implement the following functionality that will go through each element with the class name "module" and run its respective "decorate" function, which is simply associated by the module specified in the "data-module" html attribute.<br />
<br />
<pre><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;">(</span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">function</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">() {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> define([</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">'require'</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">, </span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">'jquery'</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">], </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">function</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> (require, $) {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
</div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">var</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> decorator = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">function</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> ($target) {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">var</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> moduleName = $target.dataset[</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">'module'</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">];<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
</div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> require([moduleName], </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">function</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> (module) {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">var</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> data;<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">if</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> ($.isFunction(module.decorate)) {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> data = $target.data();<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">delete</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> data.module;<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">return</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> module.decorate($target, data);<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> });<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> };<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
</div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">function</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> () {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> $(</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">".module"</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">).each(</span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">function</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> () {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">return</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;"> decorator($(</span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">this</span><span style="background: white; font-family: Consolas; font-size: 9.5pt;">));<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> });<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> };<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background: white; font-family: Consolas; font-size: 9.5pt;"> });<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="background: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">})</span><span style="font-family: Consolas; font-size: 9.5pt; line-height: 115%;">;</span><o:p></o:p></div>
</pre>
<br />
And its as simple as that! Now for any specific html element, you can associate specific javascript functionality to apply to it without having to load large amounts of javascript files unnecessarily.<br />
<br />
<br />
<br /></div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com7tag:blogger.com,1999:blog-6898784454956902947.post-52708484984797628072014-03-10T14:30:00.000+00:002014-04-29T08:39:36.918+00:00Lazy Loading in C# - Performance vs Memory<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="WordSection1">
<div class="MsoNormal">
<span style="font-family: inherit;">After a long deliberating discussion with a couple of work colleagues about the pros and cons of lazy loading compared to eager loading, a few interesting conclusions arose.</span></div>
<div class="MsoNormal">
<span style="font-family: inherit;"><br /></span></div>
<div class="MsoNormal">
<span style="font-family: inherit;">Many programmers often tend to fall into the trap of instantiating all their objects during the initialization of another parent object (either in the constructor or otherwise).</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
<pre><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">public</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">class</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">AccountController</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> : </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">BaseContoller</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">UserBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _userFunctions = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">null</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">;<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">RoleBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _roleFunctions = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">null</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">;<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">PermissionBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _permissionsFunctions = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">null</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">;<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">MailoutBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _mailoutFunctions = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">null</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">;<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
</div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">public</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> AccountController()<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _userFunctions = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">new</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">UserBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">();<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _roleFunctions = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">new</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">RoleBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">();<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _permissionsFunctions = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">new</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">PermissionBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">();<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _mailoutFunctions = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">new</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">MailoutBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">();<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
</div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">public</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">ActionResult</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> Index()<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">return</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> View();<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
</pre>
<div class="MsoNormal">
<br /></div>
<span style="font-family: inherit;">For windows applications this might be beneficial in that the initial load time may vary, however the rest of the user's experience is very smooth because everything has been loaded already.</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">However, for much larger projects this can seriously affect the processing of an application in terms of memory and initial loading. Imagine you had to create a new instance </span>of <span style="font-family: Courier New, Courier, monospace;">AccountController</span> just <span style="font-family: inherit;">to call one function (e.g. </span><span style="font-family: Courier New, Courier, monospace;">Index()</span><span style="font-family: inherit;">) and all that the function did was return an </span><span style="font-family: Courier New, Courier, monospace;">AcionResult</span><span style="font-family: inherit;"> object just as MVC does on the initial load of the default page. Moreover, web pages specifically have a decoupling of server side code to the client-side rendered HTML page; so memory is eventually released after a Response is complete. However, that means that th</span>e <span style="font-family: Courier New, Courier, monospace;">AccountController</span> class will b<span style="font-family: inherit;">e instantiated every time a request is made to the </span><span style="font-family: Courier New, Courier, monospace;">Index()</span><span style="font-family: inherit;"> function. Imagine 15,000 users make a request from their browser to the same function </span>at the same time.<br />
The Garbage Collector may not have enough time to clean up the unused functions.<br />
<br />
Some developers could write that off quite easily when they are under the impression that production servers are enormous beasts with infinite amount of RAM. That may be true to an extent, but when you use only one production server to host multiple sites, each of which can have many thousands of users, memory can be easily chewed up. You may have a cloud solution for this, and it could be as simple as "upping" the memory in cloud management, but that could get very expensive very fast.<br />
<br />
Lazy loading is the best solution to solve this problem.<br />
<br />
<pre><div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">public</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">class</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">AccountController</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> : </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">BaseContoller</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">UserBusinessFunctions </span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">_userFunctions = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">null</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">;<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">RoleBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _roleFunctions = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">null</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">;<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">PermissionBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _permissionsFunctions = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">null</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">;<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">MailoutBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _mailoutFunctions = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">null</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">;<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
</div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">public</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> AccountController()<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
</div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">public</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">ActionResult</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> Index()<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">return</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> View();<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
</div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">public</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">UserBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> UserFunctions<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">get</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">if</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> (_userFunctions == </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">null</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">)<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _userFunctions = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">new</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">UserBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">();<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">return</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _userFunctions;<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
</div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">public</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">RoleBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> RoleFunctions<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">get</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">if</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> (_roleFunctions == </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">null</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">)<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _roleFunctions = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">new</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">RoleBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">();<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">return</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _roleFunctions;<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
</div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">public</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">PermissionBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> PermissionsFunctions<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">get</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">if</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> (_permissionsFunctions == </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">null</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">)<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _permissionsFunctions = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">new</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">PermissionBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">();<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">return</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _permissionsFunctions;<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
</div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">public</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">MailoutBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> MailoutFunctions<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">get</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">if</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> (_mailoutFunctions == </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">null</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">)<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _mailoutFunctions = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">new</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">MailoutBusinessFunctions</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;">();<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; mso-highlight: white;">return</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> _mailoutFunctions;<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="margin-bottom: 0.0001pt;">
<span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
</pre>
<div class="MsoNormal">
<br /></div>
<span style="font-family: inherit;"><br /></span>As shown above, there is no longer any instantiation made in the constructor. This means that for every call to the <span style="font-family: Courier New, Courier, monospace;">Index()</span> function, there will no longer be any unnecessary allocations in memory to unused objects just to return an <span style="font-family: Courier New, Courier, monospace;">ActionResult</span> object.<br />
<br />
What that also means is, if you actually want to use one of the business function objects in the example above, then rather than using the <span style="font-family: Courier New, Courier, monospace;">_userFunctions</span> variable, you would have to access the property instead:<br />
<br />
<pre><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-ansi-language: EN-AU; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: white;">User</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt; line-height: 115%;"> user = </span><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-ansi-language: EN-AU; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: white;">this</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">.UserFunctions.GetUser(id);</span>
</pre>
</div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="font-family: inherit;">There is also another page that elaborates on the new C# 4.0 feature of "Lazy Initialization" which is slightly different to the standard "Lazy Loading" pattern mentioned above.<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: inherit;"><br /></span></div>
<div class="MsoNormal">
<span style="font-family: inherit;"><a href="http://softarchitect.wordpress.com/2010/08/14/software-architecture-c-sharp-4-0-new-feature-lazy-initialization/">http://softarchitect.wordpress.com/2010/08/14/software-architecture-c-sharp-4-0-new-feature-lazy-initialization/</a><o:p></o:p></span></div>
<div class="MsoNormal">
<span style="font-family: inherit;"><br /></span></div>
<div class="MsoNormal">
<span style="font-family: inherit;">I can see advantages in all three of the scenarios mentioned above:</span></div>
<div class="MsoNormal">
</div>
<ul style="text-align: left;">
<li><span style="font-family: inherit; text-indent: -18pt;"><i>Initializing all in the constructor</i> - good for Silverlight apps where performance is vital over memory usage on the client side. What that means is that there will be more time loading the Silverlight progress bar at the beginning while the application loads the objects in its memory resources, but the rest of the experience is seamless from then on.</span></li>
<li><span style="font-family: inherit; text-indent: -18pt;"><i>Lazy loading in the property</i> - good for servers that have multiple web applications/web services/win services running where slight performance trade-offs for valuable memory is important.</span></li>
<li><span style="font-family: inherit;"><i>C#4.0 "Lazy Initializing"</i> - good for examples where you instantiate a bunch of objects before entering a large loop, but want the loop to start as soon as possible.</span></li>
</ul>
<span style="font-family: inherit;"><br /></span>
<br />
<div class="MsoNormal">
<span style="font-family: inherit;"><o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
</div>
</div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com1tag:blogger.com,1999:blog-6898784454956902947.post-30904010228792471962014-01-14T05:18:00.000+00:002014-03-12T00:07:44.477+00:00DateTime.Parse vs DateTime.ParseExact for culture issues<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: Calibri; font-size: x-small;"><span style="font-size: 11pt;"> </span></span><br />
<div>
<span style="font-family: Calibri; font-size: 11pt;">I'm sure most of us </span><span style="font-family: Calibri; font-size: 15px;">savvy</span><span style="font-family: Calibri; font-size: 11pt;"> .NET developers have come across the dreaded issues that surround the inconsistent standards when working with </span><span style="font-family: Calibri; font-size: 15px;">multilingual</span><span style="font-family: Calibri; font-size: 11pt;"> / multicultured applications. Namely the sites that provide the opportunity for a user to change their language and/or culture for their convenience.</span><br />
<span style="font-family: Calibri; font-size: 11pt;"><br /></span>
<span style="font-family: Calibri;"><span style="font-size: 11pt;">Dealing with different languages is one thing, but that in itself isn't so bad because it mainly deals with a </span><span style="font-size: 15px;">resource</span><span style="font-size: 11pt;"> file of some sort that allows mapping of one word (or a sentence) to a different word of a different language. The site can then replace those words where it finds them.</span></span><br />
<span style="font-family: Calibri;"><span style="font-size: 11pt;"><br /></span></span>
<span style="font-family: Calibri;"><span style="font-size: 11pt;">Dealing with data, and specifically dates on the other hand, is a completely different story.</span></span><br />
<span style="font-family: Calibri;"><span style="font-size: 11pt;"><br /></span></span>
<span style="font-family: Calibri;"><span style="font-size: 11pt;">Seriously, why did the Americans have to come up with a date</span></span><span style="font-family: Calibri; font-size: 11pt;"> format that doesn't even conform to itself!? </span><span style="font-family: Calibri;"><span style="font-size: 11pt;">Generally speaking, numbering systems work by having the larger metric unit at the front, then the second metric unit succeeding it, and so on (e.g. $15,432,67). </span></span><span style="font-family: Calibri; font-size: 11pt;">Naturally you would expect the date structure to be the same so that the military format (<i>yyyy/MM/dd</i>) would be found wherever we deal with dates.</span><br />
<span style="font-family: Calibri; font-size: 11pt;"><br /></span>
<span style="font-family: Calibri; font-size: 11pt;">Even though the Australian format doesn't conform to this, it at least has consistency by having the reverse, whereby the smallest metric is found at the beginning of the sequence (<i>dd/MM/yyyy</i>).</span><br />
<span style="font-family: Calibri; font-size: 11pt;"><br /></span>
<span style="font-family: Calibri; font-size: 11pt;">The U.S. format, however, is an influence of their own language. It comes purely from a cultural perspective where they are naturally inclined to say "February the 15th" rather than "the 15th of February". That has lead to the demise of their date format which logically makes no sense (<i>MM/dd/yyyy</i>).</span><br />
<span style="font-family: Calibri; font-size: 11pt;"><br /></span>
<span style="font-family: Calibri; font-size: 11pt;">Regardless of the format, we are still faced with the dilemma of having to interpret this in code so that our data isn't corrupted by the front-end client machine's culture, and the server's culture, when the server-side code deals with the data.</span><br />
<span style="font-family: Calibri; font-size: 11pt;"><br /></span>
<span style="font-family: Calibri; font-size: 11pt;">You may find that you have a JQuery date picker that can help you "set the culture" as shown below:</span><br />
<span style="font-family: Calibri; font-size: 11pt;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYBek8SrHsuoghzVEGFBvOkJtYQCW0P0V4EIL7fVmIMC7gAfnkhh8_iNppYMU1l1Rf3AFrghktLxyHgwd3024q-J8srwBrXnBiJbLBgGfmjXFTpG60lCVz-14rKt2TPNAaPDwswQBcnHn7/s1600/DatePickerUS.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"></a><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYBek8SrHsuoghzVEGFBvOkJtYQCW0P0V4EIL7fVmIMC7gAfnkhh8_iNppYMU1l1Rf3AFrghktLxyHgwd3024q-J8srwBrXnBiJbLBgGfmjXFTpG60lCVz-14rKt2TPNAaPDwswQBcnHn7/s1600/DatePickerUS.png" /></div>
<div>
<pre><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">$(</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-ansi-language: EN-AU; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: white;">".datepicker"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">).datepicker({ dateFormat: </span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-ansi-language: EN-AU; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: white;">"dd/mm/yy"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt; line-height: 115%;"> });</span></pre>
</div>
<span style="font-family: Calibri; font-size: 11pt;"><br /></span>
<br />
<span style="font-family: Calibri; font-size: 11pt;">The problem would arise once the user clicks submit and hits a server with the "en-US" culture.</span><br />
<span style="font-family: Calibri; font-size: 11pt;">The moment you write the following code, you will face some issues:</span><br />
<span style="font-family: Calibri; font-size: 11pt;"><br /></span>
<br />
<pre><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-ansi-language: EN-AU; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: white;">var</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt; line-height: 115%;"> date = </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-ansi-language: EN-AU; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: white;">DateTime</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">.Parse(Request.Form[</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-ansi-language: EN-AU; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: white;">"date"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">]);</span></pre>
<span style="font-family: Calibri; font-size: 11pt;"><br /></span>
<span style="font-family: Calibri; font-size: 11pt;">One of two problems will arise:</span><br />
<ul style="text-align: left;">
<li><span style="font-family: Calibri;"><span style="font-size: 15px;">the date conforms to either en-AU or en-US because the "day" is not greater than 12, and therefore will not complain when translating it into a month part.</span></span></li>
<li><span style="font-family: Calibri;"><span style="font-size: 15px;">a FormatException will occur because the day is considered month and is invalid if greater than 12.</span></span></li>
</ul>
<div>
<span style="font-family: Calibri;"><span style="font-size: 15px;">The way to work around this so as to Parse the date into the current culture of the server, but based on the culture you are expecting the string to be coming in from the client side.</span></span></div>
<div>
<span style="font-family: Calibri;"><span style="font-size: 15px;"><br /></span></span></div>
<div>
<pre><span style="background: white; color: blue; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-ansi-language: EN-AU; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: white;">var</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt; line-height: 115%;"> date = </span><span style="background: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-ansi-language: EN-AU; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: white;">DateTime</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">.ParseExact(Request.Form[</span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-ansi-language: EN-AU; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: white;">"date"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">], </span><span style="background: white; color: #a31515; font-family: Consolas; font-size: 9.5pt; line-height: 115%; mso-ansi-language: EN-AU; mso-bidi-language: AR-SA; mso-fareast-font-family: Calibri; mso-fareast-language: EN-US; mso-fareast-theme-font: minor-latin; mso-highlight: white;">"dd/MM/yyyy"</span><span style="background-color: white; background-position: initial initial; background-repeat: initial initial; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">, </span><span style="background-color: white; color: #2b91af; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">CultureInfo</span><span style="background-color: white; font-family: Consolas; font-size: 9.5pt; line-height: 115%;">.CurrentCulture);</span></pre>
</div>
<div>
<span style="font-family: Calibri;"><span style="font-size: 15px;"><br /></span></span></div>
<div>
<span style="font-family: Calibri;"><span style="font-size: 15px;">What this does is define the structure of the string that is passed into the first parameter, based on the format of the second parameter. Once it maps the string exactly as defined in the second parameter, it can then convert it successfully to a DateTime type based on the third parameter. The CurrentCulture can define the timezone as well as the format, and any other relevant metadata.</span></span></div>
<br />
<span style="font-family: Calibri; font-size: 11pt;"><br /></span></div>
</div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0tag:blogger.com,1999:blog-6898784454956902947.post-34651657218783703982012-08-20T23:49:00.000+00:002014-02-26T00:04:35.615+00:00Orchard CMS - Top 10 "Good-To-Knows"<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="MsoNormal">
Currently at <a href="http://resonatesolutions.com.au/">http://resonatesolutions.com.au</a> I’ve had the opportunity to delve into the latest release of Orchard CMS (version 1.4), however working with customising certain components can be quite cumbersome, especially when the eBooks and documentation on Orchard is quite thin. Most of my success has come from reverse engineering some of the core modules that are provided in the open source version of the install. The following are some of the issues that have provided time consuming problem-solving exercises:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b>1. Directory Listing issue<o:p></o:p></b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Quite frequently I used to get this error when starting my orchard application:<o:p></o:p></div>
<pre><h2>
<i><span style="color: maroon; font-family: "Verdana","sans-serif"; font-size: 14.0pt; font-weight: normal;">Directory Listing -- /OrchardLocal/</span></i><span style="color: maroon; font-family: "Verdana","sans-serif"; font-size: 14.0pt; font-weight: normal;"><o:p></o:p></span></h2>
<div align="center" class="MsoNormal" style="text-align: center;">
<hr align="center" noshade="" size="1" style="color: black;" width="100%" />
</div>
<span style="color: black; font-family: "Lucida Console"; font-size: 8.5pt;"> Monday, June 25, 2012 12:51 PM <dir> <a href="http://localhost:30320/OrchardLocal/App_Data/">App_Data</a></span>
<span style="color: black; font-family: "Lucida Console"; font-size: 8.5pt;"> Monday, July 09, 2012 11:26 AM <dir> <a href="http://localhost:30320/OrchardLocal/bin/">bin</a></span>
<span style="color: black; font-family: "Lucida Console"; font-size: 8.5pt;"> Wednesday, May 30, 2012 03:35 PM <dir> <a href="http://localhost:30320/OrchardLocal/Config/">Config</a></span>
<span style="color: black; font-family: "Lucida Console"; font-size: 8.5pt;"> Wednesday, May 30, 2012 03:37 PM <dir> <a href="http://localhost:30320/OrchardLocal/Core/">Core</a></span>
<span style="color: black; font-family: "Lucida Console"; font-size: 8.5pt;"> Tuesday, May 22, 2012 12:58 PM 103 <a href="http://localhost:30320/OrchardLocal/Global.asax">Global.asax</a></span>
<span style="color: black; font-family: "Lucida Console"; font-size: 8.5pt;"> Tuesday, May 22, 2012 12:58 PM 2,124 <a href="http://localhost:30320/OrchardLocal/Global.asax.cs">Global.asax.cs</a></span>
<span style="color: black; font-family: "Lucida Console"; font-size: 8.5pt;"> Thursday, June 21, 2012 10:23 PM <dir> <a href="http://localhost:30320/OrchardLocal/Media/">Media</a></span>
<span style="color: black; font-family: "Lucida Console"; font-size: 8.5pt;"> Friday, July 06, 2012 06:55 PM <dir> <a href="http://localhost:30320/OrchardLocal/Modules/">Modules</a></span>
<span style="color: black; font-family: "Lucida Console"; font-size: 8.5pt;"> Sunday, July 08, 2012 01:50 PM <dir> <a href="http://localhost:30320/OrchardLocal/obj/">obj</a></span>
<span style="color: black; font-family: "Lucida Console"; font-size: 8.5pt;"> Thursday, June 28, 2012 12:42 PM 13,574 <a href="http://localhost:30320/OrchardLocal/Orchard.Web.csproj">Orchard.Web.csproj</a></span>
<span style="color: black; font-family: "Lucida Console"; font-size: 8.5pt;"> Thursday, June 28, 2012 12:42 PM 1,170 <a href="http://localhost:30320/OrchardLocal/Orchard.Web.csproj.user">Orchard.Web.csproj.user</a></span>
<span style="color: black; font-family: "Lucida Console"; font-size: 8.5pt;"> Wednesday, May 30, 2012 03:35 PM <dir> <a href="http://localhost:30320/OrchardLocal/Properties/">Properties</a></span>
<span style="color: black; font-family: "Lucida Console"; font-size: 8.5pt;"> Tuesday, May 22, 2012 12:59 PM 442 <a href="http://localhost:30320/OrchardLocal/Refresh.html">Refresh.html</a></span>
<span style="color: black; font-family: "Lucida Console"; font-size: 8.5pt;"> Monday, July 02, 2012 01:36 PM <dir> <a href="http://localhost:30320/OrchardLocal/Themes/">Themes</a></span>
<span style="color: black; font-family: "Lucida Console"; font-size: 8.5pt;"> Friday, June 22, 2012 04:26 PM 9,177 <a href="http://localhost:30320/OrchardLocal/Web.config">Web.config</a><o:p></o:p></span></pre>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
After pulling my hair out trying to find out whether this was an IIS issue or whether it was a file permission issue with the ASPNET account, I found that the problem actually lay with an exception that had occurred <i>before</i> Orchard’s URL rewriting had kicked in, thus allowing the site to land in the default directory which incidentally didn’t have a default file associated with it.<o:p></o:p></div>
<div class="MsoNormal">
Even if you were to navigate to a bookmarked URL it would give the notorious yellow screen of death with the 404 server error:<o:p></o:p></div>
<h2 style="background: white;">
<i><span style="color: maroon; font-family: "Verdana","sans-serif"; font-size: 14.0pt; font-weight: normal;">The resource cannot be found.</span></i><span style="color: maroon; font-family: "Verdana","sans-serif"; font-size: 14.0pt; font-weight: normal;"><o:p></o:p></span></h2>
<div class="MsoNormal">
In order to solve this problem you will have to go to the log file to find the actual error found in /App_Data/Logs/orchard-error-yyyy.MM.dd.log<o:p></o:p></div>
<div class="MsoNormal">
The information provided there is the exact stack trace of your code, to help you solve your problem and just like magic the site will start working again.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
If only Orchard handled this a lot better…<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b>2. Changing a property name of a ContentPartRecord in your Custom Module<o:p></o:p></b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Another issue relates to Orchard’s choice of ORM, NHibernate and its caching capabilities.<o:p></o:p></div>
<div class="MsoNormal">
Say, for example, you had the following ContentPartRecord in your Models folder of your Custom Module:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<pre><div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">public</span> <span style="color: blue;">class</span> <span style="color: #2b91af;"> PulseLookupRecord</span> : <span style="color: #2b91af;">ContentPartRecord</span><o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">public</span> <span style="color: blue;">virtual</span> <span style="color: blue;"> string</span> ConnectionString { <span style="color: blue;">get</span>; <span style="color: blue;"> set</span>; }<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
</pre>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<div class="MsoNormal">
If we were to rename this to ConnectionString1, then rename the database column that relates to this, this should work fine if we rebuild our module and start the Orchard web application.<o:p></o:p></div>
<div class="MsoNormal">
However the application starts we will notice that the field that this property is associated with will not be rendered, and Orchard’s notorious silent errors will have vital information that we have missed. When looking at the log file we notice the following error message:<o:p></o:p></div>
<h2 style="background: white;">
<i><span style="color: maroon; font-family: "Verdana","sans-serif"; font-size: 14.0pt; font-weight: normal;">NHibernate.PropertyNotFoundException: Could not find a getter for property 'ConnectionString1' in class 'Pulse.Lookup.Models.PulseLookupRecord'<o:p></o:p></span></i></h2>
<div class="MsoNormal">
We would assume that since we have updated all the required field properties that NHibernates mapping should just work as it should, however the problem exists because NHibernate’s mapping is actually cached in a file stored in \App_Data\Sites\Woolworths\mappings.bin.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Simply removing this file will force Orchard to rebuild its NHibernate mapping and everything will then work fine.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b>3. Accessing the connection string dynamically<o:p></o:p></b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
The only place that you can find the connection string that was set initially when cooking a new site or tenant is in a text file found in the /App_Data/Sites/[Default]/Settings.txt file.<o:p></o:p></div>
<div class="MsoNormal">
However it would be extremely inpractical to start using System.IO to manipulate and read the line that contains the connection string line from within that page to use it.<o:p></o:p></div>
<div class="MsoNormal">
There is another place, however, where you can retrieve the connection string dynamically via your module’s code however the biggest problem is that the _session field that contains the ConnectionString property is actually a private field found in NHibernate’s HqlQuery type. This is how you can retrieve the Session from which you can subsequently retrieve the ConnectionString property:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<pre><div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">private</span> NHibernate.ISession GetPrivateSession(<span style="color: #2b91af;">IHqlQuery</span> query)<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">return</span> (NHibernate.ISession)((<span style="color: #2b91af;">DefaultHqlQuery</span>)Services.ContentManager.HqlQuery()).GetType()<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> .GetField(<span style="color: #a31515;">"_session"</span>, <span style="color: #2b91af;">BindingFlags</span>.NonPublic | <span style="color: #2b91af;"> BindingFlags</span>.Instance).GetValue(query);<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
</pre>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b>4. Creating a custom site-wide setting<o:p></o:p></b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
There are literally two lines of code you will need to implement a site-wide setting after creating a very standard part.<o:p></o:p></div>
<div class="MsoNormal">
Add an extra filter to your handler for your part that is associated with the “Site” Content Type:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<pre><div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">public</span> <span style="color: blue;">class</span> <span style="color: #2b91af;"> PulseLookupHandler</span> : <span style="color: #2b91af;">ContentHandler</span><o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">public</span> PulseLookupHandler(<span style="color: #2b91af;">IRepository</span><<span style="color: #2b91af;">PulseLookupRecord</span>> repository)<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="background: yellow; mso-highlight: yellow;">Filters.Add(<span style="color: blue;">new</span> <span style="color: #2b91af;">ActivatingFilter</span><<span style="color: #2b91af;">PulseLookupPart</span>>(<span style="color: #a31515;">"Site"</span>));</span><o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> Filters.Add(<span style="color: #2b91af;">StorageFilter</span>.For(repository));<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
</pre>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<div class="MsoNormal">
“Site” Content Type isn’t actually visible in the Content Type list in the orchard admin, rather is found in “Settings” > “General”.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
And then to actually access the property that has been created you will need the following line which can be written anywhere in the site that uses the “Service” object:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<pre><div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">public</span> <span style="color: blue;">class</span> <span style="color: #2b91af;"> PulseLookupFieldDriver</span> : <span style="color: #2b91af;">ContentFieldDriver</span><<span style="color: #2b91af;">PulseLookupField</span>><o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">public</span> <span style="color: #2b91af;">IOrchardServices</span> Services { <span style="color: blue;">get</span>; <span style="color: blue;">set</span>; }<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
</div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">public</span> PulseLookupFieldDriver(<span style="color: #2b91af;">IOrchardServices</span> services)<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> Services = services;<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
</div>
<div class="MsoNormal" style="text-autospace: none;">
</div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">protected</span> <span style="color: blue;">override</span> <span style="color: #2b91af;">DriverResult</span> Editor(<span style="color: #2b91af;">ContentPart</span> part, <span style="color: #2b91af;">PulseLookupField</span> field, <span style="color: blue;"> dynamic</span> shapeHelper)<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="background: yellow; color: blue; mso-highlight: yellow;">var</span><span style="background: yellow; mso-highlight: yellow;"> siteSettings = Services.WorkContext.CurrentSite.As<<span style="color: #2b91af;">PulseLookupPart</span>>();</span><o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
</pre>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<div class="MsoNormal">
<b>5. Access Denied when installing a Module<o:p></o:p></b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
This again is a misleading error message that does not explain the issue. Apparently when a module is downloaded it can only be installed via the <i>Default</i> Orchard site, not one of its Tenant instances. It would be more appropriate if that option was removed completely from a Tenant instance or at least provide a more relevant error message that explicitly explains an Orchard-specific issue.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b>6. Including a custom module in your Orchard.Web causes runtime exceptions<span style="color: red;"><o:p></o:p></span></b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
The biggest issue with trying to debug a module is when you decide to include the module files within the Orchard Web project (which naturally resides in the Modules folder). This will allow you to debug up until a certain point where one of your custom types are being referenced, then the following error message will get thrown in your face:<o:p></o:p></div>
<h2 style="background: white;">
<i><span style="color: maroon; font-family: "Verdana","sans-serif"; font-size: 14.0pt; font-weight: normal;">The type 'Pulse.Lookup.ViewModels.PulseLookupFieldViewModel' exists in both '..\Pulse.Lookup.dll' and '..\Orchard.Web.DLL'<o:p></o:p></span></i></h2>
<div class="MsoNormal">
The main reason for this is that Orchard actually has a dynamic compiler that locates your .csproj file and compiles it on the fly, essentially creating a dll and storing it physically in the \Orchard.Web\App_Data\Dependencies\ folder to be used within the Orchard CMS. That that would in effect cause a problem if you have already compiled the module within the Orchard web project already. Even though during compile time there is no issues, the custom type you have created would be embedded within the Orchard.Web.dll. That means that when Orchard also compiles the module on the fly during runtime another dll would be created specifically for your module, thus creating the same object type with the same namespace in two separate dll and this is where the problem lies.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
The best way to work around this is to actually include your project as a Web Application under the “Modules” Solution folder within the solution itself (and not include the files in the Orchard.Web) folder. Attaching this Web Application as a project reference (by selecting the .csproj file) effectively attaches the debugger to the compiled version of that project which is bound by the Web application added in your Orchard solution, thus allowing breakpoints to actually work in the code files of your module within the Orchard solution.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal" style="text-autospace: none;">
<b>7. Web Server crashes after calling </b><b><span style="font-family: Consolas; font-size: 9.5pt;">updater.TryUpdateModel </span> In the Driver’s Editor function<o:p></o:p></b></div>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<div class="MsoNormal">
When creating a custom field and having an underling ViewModel to display supporting information for that field, there is an issue upon posting via the Edit form which calls the overridden Editor function in the Driver. Once the <span style="font-family: Consolas; font-size: 9.5pt;">updater.TryUpdateModel</span> function attempts to update the custom field, it will successfully update it in memory then returns from the function which then fires an underlying Orchard events that attempt to update the ContentItemVersionRecord table in the Orchard database. This ends up hanging then crashing the web server that hosts the Web application.<o:p></o:p></div>
<div class="MsoNormal">
After looking at the Event log to find out what the actual error was that crashed the web server the following .NET error was found:<o:p></o:p></div>
<h2 style="background: white;">
<i><span style="color: maroon; font-family: "Verdana","sans-serif"; font-size: 14.0pt; font-weight: normal;">Exception: System.InvalidOperationException<br /> Message: Operation is not valid due to the current state of the object.<o:p></o:p></span></i></h2>
<div class="MsoNormal">
Basically what this is trying to say is that the field’s data could not be committed to the database because there was no update state on the record to allow for a database update. You will need to tell Orchard that there has been a change made on the field and that is by flushing the current state of the content via the content manager using the following line of code:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<pre><div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">public</span> <span style="color: blue;">class</span> <span style="color: #2b91af;"> PulseLookupFieldDriver</span> : <span style="color: #2b91af;">ContentFieldDriver</span><<span style="color: #2b91af;">PulseLookupField</span>><o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">public</span> <span style="color: #2b91af;">IOrchardServices</span> Services { <span style="color: blue;">get</span>; <span style="color: blue;">set</span>; }<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
</div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">public</span> PulseLookupFieldDriver(<span style="color: #2b91af;">IOrchardServices</span> services)<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> Services = services;<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
</div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">protected</span> <span style="color: blue;">override</span> <span style="color: #2b91af;">DriverResult</span> Editor(<span style="color: #2b91af;">ContentPart</span> part, <span style="color: #2b91af;">PulseLookupField</span> field, <span style="color: #2b91af;"> IUpdateModel</span> updater, <span style="color: blue;">dynamic</span> shapeHelper)<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">if</span> (updater.TryUpdateModel(field, GetPrefix(field, part), <span style="color: blue;">null</span>, <span style="color: blue;">null</span>))<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="background: yellow; mso-highlight: yellow;">Services.ContentManager.Flush();</span><o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
</div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">return</span> Editor(part, field, shapeHelper);<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
</pre>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<div class="MsoNormal">
<b>8. Opening a separate database connection within Orchard’s current session<o:p></o:p></b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
When running two concurrent database connections and exceptions will be thrown via SQL Server and the error message will be as follows:<o:p></o:p></div>
<h2 style="background: white;">
<i><span style="color: maroon; font-family: "Verdana","sans-serif"; font-size: 14.0pt; font-weight: normal;">System.Data.SqlClient.SqlException: MSDTC on server 'localhost' is unavailable<o:p></o:p></span></i></h2>
<div class="MsoNormal">
The solution to this is as simple as starting the following service on the server: <i>“Distributed Transaction Coordinator”. </i>This can be achieved by going to Administrative Tools > Component Services > Expanding to "My Computer" > Expanding "Distributed Transaction Coordinator" > Right-Click "Local DTC" > On the Security Tab and checking the "Network DTC Access" and the "Allow Remote Clients" as well as the "Allow Remote Administration" checkboxes.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b>9. Custom fields visible in Projection Queries Filtering<o:p></o:p></b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Following the online tutorials on how to create a custom field, it was pretty simple adding it to a Content Type as shown below:<o:p></o:p></div>
<div class="MsoNormal">
<span style="mso-fareast-language: EN-AU;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyAMDTseIRUV71y6k8F3eGO0smAq_rZvLxqWpBOH-DHYstqGjrO8TxSh2HHQm2CYpWfyzumY78nOSQYBw7Z2tmQvf20TBQkcKvnMmbNWjj_6CWEjnkGIlwSx8knku7UIB04qgTMuxnov9r/s1600/image001-766934.png"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyAMDTseIRUV71y6k8F3eGO0smAq_rZvLxqWpBOH-DHYstqGjrO8TxSh2HHQm2CYpWfyzumY78nOSQYBw7Z2tmQvf20TBQkcKvnMmbNWjj_6CWEjnkGIlwSx8knku7UIB04qgTMuxnov9r/s320/image001-766934.png" id="BLOGGER_PHOTO_ID_5778905844860680946" /></a></span><o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
However when you want to use the inbuilt Query module (which is part of the Projection Module), it doesn’t get displayed by default when wanting to add filtering. In other words what we hope to achieve is to show the below when you want to add a filter:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<span style="mso-fareast-language: EN-AU;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjY3Y7kqZ0o67xmp8Kous-LLoChVZjMvQGUa4CXrYUG2jxYtCAKJZp8ozdeGsfU0FPU0SyrGJufqmbEE8cRmoprJs3zN_TYcF8-GC6myXhCSqLjDjKX9ecASwwXc6Ew8ZpUxK4OgJz9QR5p/s1600/image002-768917.png"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjY3Y7kqZ0o67xmp8Kous-LLoChVZjMvQGUa4CXrYUG2jxYtCAKJZp8ozdeGsfU0FPU0SyrGJufqmbEE8cRmoprJs3zN_TYcF8-GC6myXhCSqLjDjKX9ecASwwXc6Ew8ZpUxK4OgJz9QR5p/s320/image002-768917.png" id="BLOGGER_PHOTO_ID_5778905848485312210" /></a></span><o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Without any documentation on how to achieve this, and after a little reverse engineering I realised that I needed to override the “Describe” function in the ContentFieldDriver class that was created for the custom field.<o:p></o:p></div>
<div class="MsoNormal">
It is as simple as writing the following in your field’s Driver:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<pre><div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;">protected</span> <span style="color: blue;">override</span> <span style="color: blue;">void</span> Describe(Orchard.ContentManagement.Handlers.<span style="color: #2b91af;">DescribeMembersContext</span> context)<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> {<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> context.Member(<span style="color: blue;">null</span>, <span style="color: blue;">typeof</span>(<span style="color: blue;">string</span>), T(<span style="color: #a31515;">"Data"</span>), T(<span style="color: #a31515;">"The data associated with the field."</span>));<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> }<o:p></o:p></span></div>
</pre>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<div class="MsoNormal" style="text-autospace: none;">
You will notice the second parameter here passes in a string type. What this in effect does fire the relevant <span style="color: #2b91af; font-family: Consolas; font-size: 9.5pt;">IFieldTypeEditor</span> (in this case the <span style="color: #2b91af; font-family: Consolas; font-size: 9.5pt;">StringFieldTypeEditor</span> which is found in the Projection Module) to add it to the FilterContext which lists the fields to filter as shown above.<span style="color: #2b91af; font-family: Consolas; font-size: 9.5pt;"><o:p></o:p></span></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<b>10. Form Posts throwing Anti Forgery Exception<o:p></o:p></b></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
This issue is actually related more so with MVC3 rather than orchard itself, however it is important to note that since Orchard uses MVC3, solving this problem is relevant for any custom module implementation that required a postback on the Display view of the module.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Within the display view of the module, if you were to simply put the following form tag :<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<pre><div class="MsoNormal" style="text-autospace: none;">
<span style="color: blue; font-family: Consolas; font-size: 9.5pt;"> <</span><span style="color: maroon; font-family: Consolas; font-size: 9.5pt;">form</span><span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: red;">action</span><span style="color: blue;">="/OrchardLocal/pending"</span> <span style="color: red;">method</span><span style="color: blue;">="post"><o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: blue; font-family: Consolas; font-size: 9.5pt;"> <</span><span style="color: maroon; font-family: Consolas; font-size: 9.5pt;">input</span><span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: red;">id</span><span style="color: blue;">="Search"</span> <span style="color: red;"> name</span><span style="color: blue;">="Search"</span> <span style="color: red;">type</span><span style="color: blue;">="text"</span> <span style="color: red;">value</span><span style="color: blue;">=""</span> <span style="color: blue;"> /><o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;"><</span><span style="color: maroon;">input</span> <span style="color: red;"> type</span><span style="color: blue;">="submit"</span> <span style="color: red;">value</span><span style="color: blue;">="Search"</span> <span style="color: blue;">/></span><o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: blue; font-family: Consolas; font-size: 9.5pt;"> </</span><span style="color: maroon; font-family: Consolas; font-size: 9.5pt;">form</span><span style="color: blue; font-family: Consolas; font-size: 9.5pt;">></span><span style="font-family: Consolas; font-size: 9.5pt;"><o:p></o:p></span></div>
</pre>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
This would result in the following exception when the submit button is pressed:<o:p></o:p></div>
<h2 style="background: white;">
<i><span style="color: maroon; font-family: "Verdana","sans-serif"; font-size: 14.0pt; font-weight: normal;">System.Web.Mvc.HttpAntiForgeryException<br /> A required anti-forgery token was not supplied or was invalid<o:p></o:p></span></i></h2>
<div class="MsoNormal">
This is because MVC3 now has a way to guard against cross-site request forgery. The best way to overcome this is to have a hidden field that verifies that you are in fact the legitimate source of the request as shown below:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<pre><div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;"><</span><span style="color: maroon;">form</span> <span style="color: red;"> action</span><span style="color: blue;">="/OrchardLocal/pending"</span> <span style="color: red;"> method</span><span style="color: blue;">="post"><o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: blue; font-family: Consolas; font-size: 9.5pt;"> <</span><span style="color: maroon; font-family: Consolas; font-size: 9.5pt;">input</span><span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: red;">id</span><span style="color: blue;">="Search"</span> <span style="color: red;"> name</span><span style="color: blue;">="Search"</span> <span style="color: red;">type</span><span style="color: blue;">="text"</span> <span style="color: red;">value</span><span style="color: blue;">=""</span> <span style="color: blue;"> /><o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;"><</span><span style="color: maroon;">input</span> <span style="color: red;"> type</span><span style="color: blue;">="submit"</span> <span style="color: red;">value</span><span style="color: blue;">="Search"</span> <span style="color: blue;">/></span><o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: blue; font-family: Consolas; font-size: 9.5pt;"> <span style="background: yellow; mso-highlight: yellow;"><</span></span><span style="background: yellow; color: maroon; font-family: Consolas; font-size: 9.5pt; mso-highlight: yellow;">input</span><span style="background: yellow; font-family: Consolas; font-size: 9.5pt; mso-highlight: yellow;"> <span style="color: red;">name</span><span style="color: blue;">="__RequestVerificationToken"</span> <span style="color: red;">type</span><span style="color: blue;">="hidden"</span> <span style="color: red;"> value</span><span style="color: blue;">="y+2p1g6OrozuycUmsWXGxi7fsmByM6Kj3EGP87FoeNzZI="</span> <span style="color: blue;">/></span></span><span style="color: blue; font-family: Consolas; font-size: 9.5pt;"><o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: blue; font-family: Consolas; font-size: 9.5pt;"> </</span><span style="color: maroon; font-family: Consolas; font-size: 9.5pt;">form</span><span style="color: blue; font-family: Consolas; font-size: 9.5pt;">></span><span style="font-family: Consolas; font-size: 9.5pt;"><o:p></o:p></span></div>
</pre>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<div class="MsoNormal">
But how do we actually create this hidden field with the correct token? It’s as simple as replacing your form tag with the following razor server-side code:<o:p></o:p></div>
<pre><div class="MsoNormal">
</div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: blue; font-family: Consolas; font-size: 9.5pt;"> </span><span style="background: yellow; font-family: Consolas; font-size: 9.5pt; mso-highlight: yellow;">@</span><span style="color: blue; font-family: Consolas; font-size: 9.5pt;">using</span><span style="font-family: Consolas; font-size: 9.5pt;"> (Html.BeginFormAntiForgeryPost())</span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: blue; font-family: Consolas; font-size: 9.5pt;"> </span><span style="font-family: Consolas; font-size: 9.5pt;">{</span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: blue; font-family: Consolas; font-size: 9.5pt;"> </span><span style="font-family: Consolas; font-size: 9.5pt;"> <span style="background: yellow; mso-highlight: yellow;">@</span>Html.TextBox(<span style="color: #a31515;">"Search"</span>)</span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: blue; font-family: Consolas; font-size: 9.5pt;"> </span><span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: blue;"><</span><span style="color: maroon;">input</span> <span style="color: red;">type</span><span style="color: blue;">="submit"</span> <span style="color: red;"> value</span><span style="color: blue;">="Search"</span> <span style="color: blue;">/></span></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: blue; font-family: Consolas; font-size: 9.5pt;"> </span><span style="font-family: Consolas; font-size: 9.5pt;">}<o:p></o:p></span></div>
</pre>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<div class="MsoNormal">
<br /></div>
</div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com2tag:blogger.com,1999:blog-6898784454956902947.post-67739197385224668742011-11-07T17:30:00.000+00:002014-03-12T00:05:47.614+00:00Reporting Services Charts using OpenQuery to write MDX<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="WordSection1">
<div class="MsoNormal">
There is currently a limitation in Reporting Services when working with MDX Cube queries.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Suppose we want to display the following report:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgM_d0-CEvr6Q2rAhTyRqCHhfqsubpDK4s1S4SsZtpLKP5K8QTTTRk9nNnjoEpqxKNpmopuW2MNEwcG_ZhROQMSvFTu3M51xqitM6d8AloDDZRDG1Rf45khJ9od5wkqhwb-rJ9tgDRRjcJJ/s1600/image005-770906.png"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgM_d0-CEvr6Q2rAhTyRqCHhfqsubpDK4s1S4SsZtpLKP5K8QTTTRk9nNnjoEpqxKNpmopuW2MNEwcG_ZhROQMSvFTu3M51xqitM6d8AloDDZRDG1Rf45khJ9od5wkqhwb-rJ9tgDRRjcJJ/s320/image005-770906.png" id="BLOGGER_PHOTO_ID_5699218325321191538" /></a><o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal" style="text-autospace: none;">
The challenge with the above chart is that the “Drivers” (i.e. NPS, Navigation, etc. found on the X-Axis) that are used for grouping are actual Measures in the Cube, and the “Days of Week” values (i.e. Monday - Sunday) are actually based off a single date Dimension.<o:p></o:p></div>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<div class="MsoNormal" style="text-autospace: none;">
In effect this means that when we create a new Dataset in the Reporting Services designer the following fields would be generated:<o:p></o:p></div>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<div class="MsoNormal" style="text-autospace: none;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTF9EMTBHTZ9Uos5O62ARzlTCtRzG6rmzu8hTVO3vkxHVTbOWfTChK576_DO4LHtm7lgbNsVZhCXnHn9Rud9_WfWlz9-TuElL4vAjYbltoK8cy8tKkENTikL6pOG_QK9xwiUbfkUv_g3R5/s1600/image008-771890.png"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTF9EMTBHTZ9Uos5O62ARzlTCtRzG6rmzu8hTVO3vkxHVTbOWfTChK576_DO4LHtm7lgbNsVZhCXnHn9Rud9_WfWlz9-TuElL4vAjYbltoK8cy8tKkENTikL6pOG_QK9xwiUbfkUv_g3R5/s320/image008-771890.png" id="BLOGGER_PHOTO_ID_5699218326148993442" /></a><o:p></o:p></div>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<div class="MsoNormal" style="text-autospace: none;">
This won’t help us group the “Drivers” in the chart because we would actually need a single column to group by.<o:p></o:p></div>
<div class="MsoNormal" style="text-autospace: none;">
So within the context options of the report the following is what we would hope to achieve:<o:p></o:p></div>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<div class="MsoNormal" style="text-autospace: none;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDEXEYMGq0u9fr1DxCf85GRn4NguRKBAT9PM09L5QK-atkYtxCuv_Ida63jazdp9n_rCPAnlVKnP9e0ZEv8qxSfch2tY2MbYpZTDhP2PbLr27KglkQoMgWJL52wLfdedxw6g5_fUmARsKn/s1600/image009-773231.png"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgDEXEYMGq0u9fr1DxCf85GRn4NguRKBAT9PM09L5QK-atkYtxCuv_Ida63jazdp9n_rCPAnlVKnP9e0ZEv8qxSfch2tY2MbYpZTDhP2PbLr27KglkQoMgWJL52wLfdedxw6g5_fUmARsKn/s320/image009-773231.png" id="BLOGGER_PHOTO_ID_5699218337635100770" /></a><o:p></o:p></div>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<div class="MsoNormal" style="text-autospace: none;">
That means that our Dataset must look like this:<o:p></o:p></div>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<div class="MsoNormal" style="text-autospace: none;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPKqSBNqrDp87BBAdS7g4i8yRz8IHHuNHvlFjVCu3yK9Np6jjDjyidUxMPKQ6Qe5jIU7Kqv1xrsN1-OfpM7b5JUuX5M4mEISD60rNliHK8rPJIrql6R8fvad-DOCXALqOFI1CnvcQraiUi/s1600/image007-774080.png"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPKqSBNqrDp87BBAdS7g4i8yRz8IHHuNHvlFjVCu3yK9Np6jjDjyidUxMPKQ6Qe5jIU7Kqv1xrsN1-OfpM7b5JUuX5M4mEISD60rNliHK8rPJIrql6R8fvad-DOCXALqOFI1CnvcQraiUi/s320/image007-774080.png" id="BLOGGER_PHOTO_ID_5699218340691446482" /></a><o:p></o:p></div>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<div class="MsoNormal" style="text-autospace: none;">
This is extremely easy to achieve in MDX where we would write the following keeping in mind that we want our Measures as rows and our single “Days Of Week” Dimension in the columns.<o:p></o:p></div>
<div class="MsoNormal" style="text-autospace: none;">
The following MDX query would provide what we are looking for:<o:p></o:p></div>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<pre><div class="MsoNormal" style="text-autospace: none;">
<span style="color: blue; font-family: 'Courier New'; font-size: 10pt;">
</span>
<span style="color: blue; font-family: 'Courier New'; font-size: 10pt;">SELECT</span><span style="font-family: 'Courier New'; font-size: 10pt;"> <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> <span style="color: blue;">NON</span> <span style="color: blue;">EMPTY</span> {<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> [Measures].[NPS], <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> [Measures].[Navigation], <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> [Measures].[Accessibility], <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> [Measures].[Availability], <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> [Measures].[Checkout], <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> [Measures].[Staff], <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> [Measures].[Cleanliness]} <span style="color: blue;">ON</span> <span style="color: blue;">ROWS</span>, <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> <span style="color: blue;">NON</span> <span style="color: blue;">EMPTY</span> <span style="color: maroon;"> CROSSJOIN</span>({ <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> ([Days Of Week].[Order Index].[Order Index] ) }, <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> { ([Days Of Week].[Day Of Week].[Day Of Week] ) }) <span style="color: blue;">ON</span> <span style="color: blue;">COLUMNS</span> <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: blue; font-family: 'Courier New'; font-size: 10pt;">FROM</span><span style="font-family: 'Courier New'; font-size: 10pt;"> [Database]<o:p></o:p></span></div>
</pre>
<div class="MsoNormal" style="text-autospace: none;">
This would return the following result:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrfR7zKkpKVQt0fxl_NUIoaDWMVGNPGn9uMJI_-y4XR06UQYxPGL5hYRVOZIIpR6FJyuaQ37q-APVD4hm3K_hzoURDd5_oRhyphenhyphenVSE1lMq5-D5IdXwcw-P6zE_FcLRpFe7O_9UXnoSpIeLbA/s1600/image003-775398.png"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrfR7zKkpKVQt0fxl_NUIoaDWMVGNPGn9uMJI_-y4XR06UQYxPGL5hYRVOZIIpR6FJyuaQ37q-APVD4hm3K_hzoURDd5_oRhyphenhyphenVSE1lMq5-D5IdXwcw-P6zE_FcLRpFe7O_9UXnoSpIeLbA/s320/image003-775398.png" id="BLOGGER_PHOTO_ID_5699218344298501826" /></a><o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
You would think that this would then naturally be simpler to bind to the Chart now that we have the correct columns to bind to our “Category Groups” in the Chart’s context option.<o:p></o:p></div>
<div class="MsoNormal">
Unfortunately, however, Reporting Services doesn't like this because having a Dimension displayed as columns would result in the following error:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYbjReOTDi7lk5mJ9oucQBhw4JI_jkpYAsKYG5KrvHeGlYHIlTdYxhv8-11F3S-scoW5OxFFMd7MM17r8mQd9D6hYM0LTDaboWmHPHRC2rmSbJsJZCQooTejxKv-MhrFemPg_dz5DG4RAZ/s1600/image011-776148.png"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYbjReOTDi7lk5mJ9oucQBhw4JI_jkpYAsKYG5KrvHeGlYHIlTdYxhv8-11F3S-scoW5OxFFMd7MM17r8mQd9D6hYM0LTDaboWmHPHRC2rmSbJsJZCQooTejxKv-MhrFemPg_dz5DG4RAZ/s320/image011-776148.png" id="BLOGGER_PHOTO_ID_5699218350193927522" /></a><o:p></o:p></div>
<div class="MsoNormal">
<i>“Could not create a list of fields for the query. Verify that you can connect to the data source and that your query syntax is correct.”<o:p></o:p></i></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
In order to solve this problem we will need to generate a Dataset using OPENQUERY.<o:p></o:p></div>
<div class="MsoNormal">
This can get tricky when working with multiple servers that host the SQL Server database and the Analysis Services database.<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
The first step would be to create a Linked Server on SQL Server. This will create a connection from SQL Server to Analysis Services so we can actually write MDX queries from a SQL Server query:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHAjk8sxUcwSTE9ZK_R1GU8H41D_35mU62fIMFZBbQROVUlPyZtcv-ZAX_zhzYEc3aXpzm8ihPa6r4lrhzdXC72zIiMRFLS0lUy2AB4HqL7MTWaNIyyKpZA1FGwfn-42M_jJY_XxCRkvia/s1600/image010-777436.png"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHAjk8sxUcwSTE9ZK_R1GU8H41D_35mU62fIMFZBbQROVUlPyZtcv-ZAX_zhzYEc3aXpzm8ihPa6r4lrhzdXC72zIiMRFLS0lUy2AB4HqL7MTWaNIyyKpZA1FGwfn-42M_jJY_XxCRkvia/s320/image010-777436.png" id="BLOGGER_PHOTO_ID_5699218353107104994" /></a><o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfkh2NaPi5sH8h0dOkX09vrnykr-PbDEr949AAyJBuFZh4V2BIhRfkvgJYY5IJaoNFqOFehRVy30Lx3XMt8POCr1jeEsn2kKyEG5S926f6oIXacnla3pDSUQOuOMQmJPWGZho2g3RFdMwL/s1600/image006-778248.png"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfkh2NaPi5sH8h0dOkX09vrnykr-PbDEr949AAyJBuFZh4V2BIhRfkvgJYY5IJaoNFqOFehRVy30Lx3XMt8POCr1jeEsn2kKyEG5S926f6oIXacnla3pDSUQOuOMQmJPWGZho2g3RFdMwL/s320/image006-778248.png" id="BLOGGER_PHOTO_ID_5699218355790469298" /></a><o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Once this is created we can now use OPENQUERY as follows:<o:p></o:p></div>
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<pre><div class="MsoNormal">
<span style="color: blue; font-family: 'Courier New'; font-size: 10pt;">
</span>
<span style="color: blue; font-family: 'Courier New'; font-size: 10pt;">SELECT</span><span style="font-family: 'Courier New'; font-size: 10pt;"> <span style="color: grey;">*</span> <span style="color: blue;">FROM</span> <span style="color: blue;"> OpenQuery</span><span style="color: grey;">(</span>[LINK]<span style="color: grey;">,</span><span style="color: red;">'</span></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: red; font-family: 'Courier New'; font-size: 10pt;">SELECT </span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: red; font-family: 'Courier New'; font-size: 10pt;"> NON EMPTY {</span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: red; font-family: 'Courier New'; font-size: 10pt;"> [Measures].[NPS], <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: red; font-family: 'Courier New'; font-size: 10pt;"> [Measures].[Navigation], <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: red; font-family: 'Courier New'; font-size: 10pt;"> [Measures].[Accessibility], <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: red; font-family: 'Courier New'; font-size: 10pt;"> [Measures].[Availability], <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: red; font-family: 'Courier New'; font-size: 10pt;"> [Measures].[Checkout], <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: red; font-family: 'Courier New'; font-size: 10pt;"> [Measures].[Staff], <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: red; font-family: 'Courier New'; font-size: 10pt;"> [Measures].[Cleanliness]} ON ROWS, <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: red; font-family: 'Courier New'; font-size: 10pt;"> NON EMPTY CROSSJOIN({ <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: red; font-family: 'Courier New'; font-size: 10pt;"> ([Days Of Week].[Order Index].[Order Index] ) }, <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: red; font-family: 'Courier New'; font-size: 10pt;"> { ([Days Of Week].[Day Of Week].[Day Of Week] ) }) ON COLUMNS <o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: red; font-family: 'Courier New'; font-size: 10pt;">FROM [Database]<o:p></o:p></span></div>
<div class="MsoNormal">
<span style="color: red; font-family: 'Courier New'; font-size: 10pt;">'</span><span style="color: grey; font-family: 'Courier New'; font-size: 10pt;">)</span><o:p></o:p></div>
</pre>
<div class="MsoNormal">
That will provide us with the following result in SQL Server Query Analyser:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_e4uXgy1b_aiNAgni0B7U1ClixtHQUFLqY-aOvvZASIJL6VKkR0ts2LCuAyqU4bhSSnxeaF4W0S9ezQpCSUAMcy51iMVO_N9pM9sGJJJfkd7RP6uSWo0qPTIwu5Pd70DCo-_-TZ0F8mYC/s1600/image012-779838.png"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_e4uXgy1b_aiNAgni0B7U1ClixtHQUFLqY-aOvvZASIJL6VKkR0ts2LCuAyqU4bhSSnxeaF4W0S9ezQpCSUAMcy51iMVO_N9pM9sGJJJfkd7RP6uSWo0qPTIwu5Pd70DCo-_-TZ0F8mYC/s320/image012-779838.png" id="BLOGGER_PHOTO_ID_5699218360508323858" /></a><o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Since Reporting Services will consider this a simple table rather than a complex MDX query, it can now be used in the report when creating a dataset in order to generate the following fields:<br />
<br />
<o:p></o:p></div>
<div class="MsoNormal">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPKqSBNqrDp87BBAdS7g4i8yRz8IHHuNHvlFjVCu3yK9Np6jjDjyidUxMPKQ6Qe5jIU7Kqv1xrsN1-OfpM7b5JUuX5M4mEISD60rNliHK8rPJIrql6R8fvad-DOCXALqOFI1CnvcQraiUi/s1600/image007-774080.png"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPKqSBNqrDp87BBAdS7g4i8yRz8IHHuNHvlFjVCu3yK9Np6jjDjyidUxMPKQ6Qe5jIU7Kqv1xrsN1-OfpM7b5JUuX5M4mEISD60rNliHK8rPJIrql6R8fvad-DOCXALqOFI1CnvcQraiUi/s320/image007-774080.png" id="BLOGGER_PHOTO_ID_5699218340691446482" /></a><o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
Naturally that means needing to map the “Monday” field to “[Days Of Week].[Order Index].&[1].[Days Of Week].[Day Of Week].&[Monday]” and so on as shown below in the Fields section:<o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTpYhUZ4kFRp45mbYf09VeqwvDUOVAem_WVAT_sitAlhiZur9hZQG-cQ88JGbHxF3qqmCoS6mD-vduGKMZ-hRNLnY_25EwkiJRYQtU1l24ya7nxnlAbckPMJkb0ysF2Uj6K8s6zD7bkkES/s1600/image013-780905.png"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTpYhUZ4kFRp45mbYf09VeqwvDUOVAem_WVAT_sitAlhiZur9hZQG-cQ88JGbHxF3qqmCoS6mD-vduGKMZ-hRNLnY_25EwkiJRYQtU1l24ya7nxnlAbckPMJkb0ysF2Uj6K8s6zD7bkkES/s320/image013-780905.png" id="BLOGGER_PHOTO_ID_5699218365216406466" /></a><o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
One important note to make is that Integrated security may seem to be the easiest security credentials to work with when connecting from, for example, a ReportViewer control in an ASP.NET site to the Reporting Services report, which then in turn connects to the SQL Server database, and then to the Analysis Services database via OPENQUERY. However this will not necessarily work since each connection along the path witll have its own Windows credentials.<o:p></o:p></div>
<div class="MsoNormal">
Ideally the Data source’s connection string in the Report Server should actually be set to a SQL login so that both the Report can work either directly from a browser or through the ReportViewer.<o:p></o:p></div>
<div class="MsoNormal">
The Linked Server settings should be set to use a specific security context for a login that has read access to the Cube. This can be set in the Linked Server as follows:<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvqVJTduvnNG9K15mAs6QPy4qnQYfu2D78SXSGjJ_F9qc1ggLavq2P3_qx3OupDSkieptFBbGlZlNYfj81crSRu5A-_nSNeOeaiWqwT4C-PyBmu6Qtx6IapIPznztKFydLL2ikq3SwaZin/s1600/image014-782471.png"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvqVJTduvnNG9K15mAs6QPy4qnQYfu2D78SXSGjJ_F9qc1ggLavq2P3_qx3OupDSkieptFBbGlZlNYfj81crSRu5A-_nSNeOeaiWqwT4C-PyBmu6Qtx6IapIPznztKFydLL2ikq3SwaZin/s320/image014-782471.png" id="BLOGGER_PHOTO_ID_5699218372768980338" /></a><o:p></o:p></div>
<div class="MsoNormal">
<br /></div>
<div class="MsoNormal">
<br /></div>
</div>
</div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0tag:blogger.com,1999:blog-6898784454956902947.post-89696801892952440452011-08-23T15:30:00.000+00:002020-02-10T09:38:12.977+00:00MSSQL recursive list in a Stored Procedure<div dir="ltr" style="text-align: left;" trbidi="on">
<div class="WordSection1">
The following scenario allows one Staff Manager to view all direct reports and their child member's direct reports recursively:
<br />
<div class="MsoNormal" style="text-autospace: none;">
<br /></div>
<pre><div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> <span style="color: blue;">WITH</span> Hierachy<span style="color: grey;">(</span>StaffNo<span style="color: grey;">,</span> ManagerStaffNo<span style="color: grey;">,</span> PreferredName<span style="color: grey;">,</span> <span style="color: blue;">Level</span><span style="color: grey;">)<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> <span style="color: blue;">AS<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="color: blue; font-family: 'Courier New'; font-size: 10pt;"> </span><span style="color: grey; font-family: 'Courier New'; font-size: 10pt;">(<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> <span style="color: blue;">SELECT</span> StaffNo<span style="color: grey;">,</span> ManagerStaffNo<span style="color: grey;">,</span> PreferredName<span style="color: grey;">,</span> 0 <span style="color: blue;">AS</span> <span style="color: blue;">Level<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> <span style="color: blue;">FROM</span> Staff s<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> <span style="color: blue;">WHERE</span> s<span style="color: grey;">.</span>StaffNo <span style="color: grey;"> =</span> @StaffNo<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
</div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> <span style="color: blue;">UNION</span> <span style="color: grey;">ALL<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-autospace: none;">
</div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> <span style="color: blue;">SELECT</span> s<span style="color: grey;">.</span>StaffNo<span style="color: grey;">,</span> s<span style="color: grey;">.</span>ManagerStaffNo<span style="color: grey;">,</span> s<span style="color: grey;">.</span>PreferredName<span style="color: grey;">,</span> sh<span style="color: grey;">.</span><span style="color: blue;">Level</span> <span style="color: grey;"> +</span> 1<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> <span style="color: blue;">FROM</span> Staff s<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> <span style="color: grey;">INNER</span> <span style="color: grey;">JOIN</span> Hierachy sh <span style="color: blue;">ON</span> s<span style="color: grey;">.</span>ManagerStaffNo <span style="color: grey;">=</span> sh<span style="color: grey;">.</span>StaffNo<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> <span style="color: grey;">)<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> <span style="color: blue;">SELECT</span> StaffNo<span style="color: grey;">,</span> ManagerStaffNo<span style="color: grey;">,</span> PreferredName<span style="color: grey;">,</span> <span style="color: blue;">Level<o:p></o:p></span></span></div>
<div class="MsoNormal" style="text-autospace: none;">
<span style="font-family: 'Courier New'; font-size: 10pt;"> <span style="color: blue;">FROM</span> Hierachy<o:p></o:p></span></div>
<div class="MsoNormal" style="text-autospace: none;">
</div>
</pre>
</div>
</div>Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0tag:blogger.com,1999:blog-6898784454956902947.post-41096175723745747372011-06-15T05:29:00.000+00:002012-02-01T04:11:59.487+00:00Dynamic Themes in MVC and handling Page_PreInit<div dir="ltr" style="text-align: left;" trbidi="on">
Another thing I learnt at <a href="http://resonatesolutions.com.au/">Resonate Solutions</a>:<br />
Generally, when you create an MVC Application the project template will provide a Master Page that can be referenced by any Views you create.<br />
That means when you want to implement generic code that executes for all page loads, typically the most logical place to put it is in the Master Page's code behind.<br />
This would be ideal if you were to implement the following code to set a theme dynamically (based on the themes in the App_Theme folder):<br />
<br />
<pre><div class="MsoNormal">
<span style="color: blue; font-family: Consolas; font-size: 9.5pt;">this</span><span style="font-family: Consolas; font-size: 9.5pt;">.Page.Theme = Request.QueryString[<span style="color: #a31515;">"Theme"</span>];</span></div>
</pre>
The page's theme, however, is processed during the initialization stage of the page's life cycle, and therefore if we need to set it, it must be done before hand i.e. in the OnPreInit event handler.<br />
The problem is that the Master Page doesn't actually have an OnPreInit event because it isn't actually a page in its own right.<br />
It is a user control that merges with the content page during the initialization phase of the page's processing, so just like a UserControl it wouldn't have an OnPreInit.<br />
So therein lies our problem. Rather than copying and pasting the above code for every single View. We would need to create a BaseViewPage that is inherited by all the Views.<br />
Take note that all Views inherit off the following class:<br />
<br />
<pre><div class="MsoNormal">
<span style="color: blue; font-family: Consolas; font-size: 9.5pt;">namespace</span><span style="font-family: Consolas; font-size: 9.5pt;"> Hub.Samples.Mvc
</span></div>
<div class="MsoNormal">
{
</div>
<div class="MsoNormal">
<span style="font-family: Consolas; font-size: 9.5pt;"><span style="color: blue;"> public</span> <span style="color: blue;">class</span> <span style="color: #2b91af;">BaseViewPage</span><T> : <span style="color: #2b91af;">ViewPage</span><T> <span style="color: blue;">where</span> T : <span style="color: blue;">class</span></span></div>
<div class="MsoNormal">
<span style="font-family: Consolas; font-size: 9.5pt;"> {</span></div>
<div class="MsoNormal">
<span style="font-family: Consolas; font-size: 9.5pt;"><span style="color: blue;"> this</span>.Page.Theme = Request.QueryString[<span style="color: #a31515;">"Theme"</span>];</span></div>
<div class="MsoNormal">
<span style="font-family: Consolas; font-size: 9.5pt;"> }</span></div>
<div class="MsoNormal">
<span style="font-family: Consolas; font-size: 9.5pt;">}</span></div>
</pre>
<br />
That also means that each View we create needs to inherit off it. This can obviously be done in the page directive as follows:<br />
<br />
<br />
<pre><div class="MsoNormal">
<span style="background-attachment: scroll; background-clip: initial; background-color: yellow; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat; font-family: Consolas; font-size: 9.5pt;"><%</span><span style="color: blue; font-family: Consolas; font-size: 9.5pt;">@</span><span style="font-family: Consolas; font-size: 9.5pt;"> <span style="color: maroon;">Page</span> <span style="color: red;">Inherits</span><span style="color: blue;">="Hub.Samples.Mvc.BaseViewPage<dynamic>"</span> <span style="background-attachment: scroll; background-clip: initial; background-color: yellow; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat;">%></span></span></div>
</pre>
</div>Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0tag:blogger.com,1999:blog-6898784454956902947.post-69799627221749222942011-03-08T04:10:00.000+00:002015-12-18T03:36:57.314+00:00Linq to Entities Count() using IQueryable<T> in Extension methods<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
Something I learnt at <a href="http://resonatesolutions.com.au/">Resonate Solutions</a> while working particularly on performance tuning of Linq To Entities when I was digging a little deeper into the IQueryable<T>.Count() method.<br />
I realised the when I was using SQL Profiler to see what the generated TSQL code was, it didn't have the equivalent COUNT(*) operator. It was actually evaluating the entire expression first then passed it back to the application so that the count could be made on the instance collection in memory, which would have a serious impact on the performance of the query because of all the data being sent back and forth.<br />
The main reason it was doing this was because the Extension Method I used was for a type IEnumerable<T> as shown below even though the instance passed in was of type IQueryable<T><br />
<br />
<pre><div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"><span style="color: blue;">public</span> <span style="color: blue;">static</span> <span style="color: blue;">int</span> PageCount</span><span style="font-family: "consolas"; font-size: 9.5pt;"><T></span><span style="font-family: "consolas"; font-size: 9.5pt;">(<span style="color: blue;">this</span> <span style="color: #2b91af;">IEnumerable</span></span><span style="font-family: "consolas"; font-size: 9.5pt;"><T></span><span style="font-family: "consolas"; font-size: 9.5pt;"><span style="color: #2b91af;"></span> list)
</span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;">{</span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: blue;">int</span> count = list.Count();</span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: green;">//rest of the code...</span></span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;">}</span></div>
</pre>
<br />
<br />
Since IQueryable<T> implements IEnumerable<T> I thought it would be good to use a "baser" Interface, but as it happened, it returned the entire collection then did a count on the inferred IEnumerable<T> instance in memory.<br />
<br />
Looking into SQL Profiler, the TSQL that gets generated contains the following:<br />
<br />
<pre><div class="MsoNormal">
<span style="color: blue; font-family: "courier new"; font-size: 10pt;">exec</span><span style="font-family: "courier new"; font-size: 10pt;"> <span style="color: maroon;">sp_executesql</span><span style="color: blue;"> </span><span style="color: red;">N'SELECT </span></span></div>
<div class="MsoNormal">
<span style="color: red; font-family: "courier new"; font-size: 10pt;">[Project2].[C1] AS [C1], </span></div>
<div class="MsoNormal">
<span style="color: red; font-family: "courier new"; font-size: 10pt;">[Project2].[C2] AS [C2], </span></div>
<div class="MsoNormal">
<span style="color: red; font-family: "courier new"; font-size: 10pt;">FROM ( SELECT </span></div>
<div class="MsoNormal">
<span style="color: red; font-family: "courier new"; font-size: 10pt;"> [Distinct1].[C1] AS [C1], </span>
</div>
<div class="MsoNormal">
<span style="color: red; font-family: "courier new"; font-size: 10pt;"> [Distinct1].[C2] AS [C2]</span></div>
</pre>
<br />
However if I were to use:<br />
<br />
<pre><div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: blue;">public</span> <span style="color: blue;">static</span> <span style="color: blue;">int</span> PageCount<T>(<span style="color: blue;">this</span> <span style="color: #2b91af;">IQueryable</span></span><span style="font-family: "consolas"; font-size: 9.5pt;"><T></span><span style="font-family: "consolas"; font-size: 9.5pt;"> list)<span style="color: #2b91af;"></span>
</span></div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;">{</span>
</div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: blue;">int</span> count = list.Count();</span></div>
</pre>
</div>
</div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;"> <span style="color: green;">//rest of the code...</span></span>
</div>
<div class="MsoNormal">
<span style="font-family: "consolas"; font-size: 9.5pt;">}</span></div>
That evaluates the TSQL with COUNT(1) which is exactly what I wanted as shown below:<br />
<pre><div class="MsoNormal">
<span style="color: blue; font-family: "courier new"; font-size: 10pt;">exec</span><span style="font-family: "courier new"; font-size: 10pt;"> <span style="color: maroon;">sp_executesql</span><span style="color: blue;"> </span><span style="color: red;">N'SELECT </span></span></div>
<div class="MsoNormal">
<span style="color: red; font-family: "courier new"; font-size: 10pt;">[GroupBy1].[A1] AS [C1]</span></div>
<div class="MsoNormal">
<span style="color: red; font-family: "courier new"; font-size: 10pt;">FROM ( SELECT </span></div>
<div class="MsoNormal">
<span style="color: red; font-family: "courier new"; font-size: 10pt;"> <span style="background-attachment: scroll; background-clip: initial; background-color: yellow; background-image: none; background-origin: initial; background-position: 0% 0%; background-repeat: repeat repeat;">COUNT(1)</span> AS [A1]</span></div>
<div class="MsoNormal">
<span style="color: red; font-family: "courier new"; font-size: 10pt;"> FROM ( SELECT DISTINCT </span></div>
<div class="MsoNormal">
<span style="color: red; font-family: "courier new"; font-size: 10pt;"> [Project1].[C1] AS [C1], </span></div>
<div class="MsoNormal">
<span style="color: red; font-family: "courier new"; font-size: 10pt;"> [Project1].[C2] AS [C2]</span>
</div>
</pre>
The main reason for this is that the inference on an interface is quite different than doing it on an abstract function or base class when everything is done in memory.<br />
When passing in the instance of an object that implements IQueryable<T> through a parameter that is of type IEnumerable<T> the instance is forced to shape itself as an IEnumerable and therefore Linq to Entities will recognise that it needs to evaluate the whole collection first rather than building the expression prior to execution.<br />
<br /></div>
Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com1tag:blogger.com,1999:blog-6898784454956902947.post-75834570636113249832010-04-25T04:05:00.001+00:002011-03-07T08:23:44.615+00:00Telerik Smarts<h3>RAD Alerts</h3>To convert the standard Javascript Alerts to the fancy RAD Alerts add this script within the <Head> tag.<br />
<br />
<pre><script type="text/javascript">
window.alert = function(string)
{
radalert(string.replace(/-/g, "<br />•"), "auto", null, "Alert");
}
</script>
</pre><br />
Add the RadWindowManager in the page and configure it with the appropriate Telerik Skin.<br />
<br />
<pre><telerik:RadWindowManager runat="server"
ID="alertWindowManager"
Skin="Default"
AutoSize="true"
Animation="None" />
</pre><br />
<h3>Themes and Demos</h3>There are also plenty of pretty cool demos of all the controls that telerik have created:<br />
<br />
<a class="external free" href="http://demos.telerik.com/aspnet-ajax/window/examples/windowmanager/defaultcs.aspx" rel="nofollow" title="http://demos.telerik.com/aspnet-ajax/window/examples/windowmanager/defaultcs.aspx">http://demos.telerik.com/aspnet-ajax/window/examples/windowmanager/defaultcs.aspx</a>Adel Helalhttp://www.blogger.com/profile/10666066501005522895noreply@blogger.com0