Thursday, December 04, 2014

TTransportException SocketTimeoutException - Hive - Azure HDInsight Emulator

While working on Hive in Azure HDInsight Emulator I was not able to create database and following exception was being thrown with all the queries:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: org.apache.thrift.transport.

To resolve the issue and assuming Hadoop installation folder is C:\hdp open hive-site.xml from:
C:\hdp\hive-....*\conf directory

Locate

  javax.jdo.option.ConnectionDriverName
  org.apache.derby.jdbc.CommonDriver
  Driver class name for a JDBC metastore


and change CommonDriver to EmbeddedDriver:

  javax.jdo.option.ConnectionDriverName
  org.apache.derby.jdbc.EmbeddedDriver
  Driver class name for a JDBC metastore


From Services restart Apache Hadoop Metastore service and retry running HQL queries. It should work not. In case issue cannot be resolved feel free to drop me an email or comment.

Thursday, July 03, 2014

ODP.NET Managed Data Access (Oracle.ManagedDataAccess.dll) with Enterprise Library

Oracle.ManagedDataAccess.dll is managed Oracle connector that doesn't require any Instant Client to be installed and offers advantage over unmanaged counterparts. It doesn't require any installation on the machine and Oracle.ManagedDataAccess.dll should be placed in execution directory or in probing directory specified in configuration.

Below are the two errors which you may receive while trying to use Oracle Managed Data Access DLL:

1. Activation error occured while trying to get instance of type Database, key ""

Resolution of the dependency failed, type = Microsoft.Practices.EnterpriseLibrary.Data.Database, name = ConnectionString.
Exception occurred while: while resolving.
Exception is: InvalidOperationException - The type Database cannot be constructed. You must configure the container to supply this value.-----------------------------------------------
At the time of the exception, the container was: Resolving  Microsoft.Practices.EnterpriseLibrary.Data.Database,ConnectionString (this occurs when wrong provider factory is used or DbProviderFactories hasn't been added to the application configuration or machine.config)


2. Configuration system failed to initialize (this occurs when configuration section doesn't have correct entry)

Solution:
To make Oracle.ManagedDataAccess.dll work with Enterprise Library few additional obvious configuration changes are required:

 Add configSections section at the top of configuration:
<configSections> <section name="oracleConnectionSettings" type="Microsoft.Practices.EnterpriseLibrary.Data.Oracle.Configuration.OracleConnectionSettings, Microsoft.Practices.EnterpriseLibrary.Data, Version=5.0.414.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" requirePermission="true" /> <section name="dataConfiguration" type="Microsoft.Practices.EnterpriseLibrary.Data.Configuration.DatabaseSettings, Microsoft.Practices.EnterpriseLibrary.Data, Version=5.0.414.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" requirePermission="true"/> <section name="oracle.manageddataaccess.client" type="OracleInternal.Common.ODPMSectionHandler, Oracle.ManagedDataAccess, Version=4.121.1.0, Culture=neutral, PublicKeyToken=89b483f429c47342" /> </configSections>


Add Db Provider Factories:

In case of ODP.NET to load appropriate factory while creating database machine.config is modified during installation and DbProviderFactories is added however with Oracle.ManagedDataAccess.dll there is no installation required so edit the application configuration file (app.config or web.config) and add following:
<system.data> <DbProviderFactories> <remove invariant="Oracle.ManagedDataAccess" /> <add name="Oracle Managed Data Provider for .NET" invariant="Oracle.ManagedDataAccess" description="Oracle Managed Data Provider for .NET" type="Oracle.ManagedDataAccess.Client.OracleClientFactory, Oracle.ManagedDataAccess, Version=4.121.1.0, Culture=neutral, PublicKeyToken=89b483f429c47342" /> </DbProviderFactories> </system.data>


Make sure to change the version and Public Key Token depending on the version being used.


Connection string for EZConnect looks like:
<connectionStrings> <add name="ConnectionString" connectionString="Data Source=//localhost:1521/Shash;User ID=scott;Password=tiger;" providerName="Oracle.ManagedDataAccess" /> </connectionStrings>

Above configuration changes allow use of Oracle.ManagedDataAccess.dll with Enterprise Library.

Monday, September 30, 2013

Calling LoadLibraryEx on ISAPI filter "C:\Windows\Microsoft.NET\Framework\v4.0.30319\aspnet_filter.dll" failed

Installed .NET Framework 1.1 for Driver caused TDoD Service to stop with error - Calling LoadLibraryEx on ISAPI filter "C:\Windows\Microsoft.NET\Framework\v4.0.30319\aspnet_filter.dll" failed.

To fix the error remove ISAPI extension from ISAPI filters added in website by .NET Fx 1.1 installer.


Tuesday, July 16, 2013

Python Libraries For Scraping

List of python scraping libraries I use to develop crawlers based on my choice of scraping library:
  1. No other than - SCRAPY - fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing
  2. UrlLib2 + Beautiful Soup - If I had to build framework from scratch this is the first choice
  3. Mechanize + Beautiful Soup - Replace UrlLib2 with Mechanize - Easy HTML form filling, any URL can be opened, not just HTTP, Automatic handling of HTTP-Equiv and Refresh, Easy link parsing and following
Please leave a feedback if you are using some other library that I should list here.

BeClasp Consulting provides Python and .NET based website scraping service and have wrote 1000's of parsers so far ranging from data crawling for Bank Accounts reconciliation, e-commerce stores or other data mining services. Drop us an email at mail@beclaspconsulting.net to know more about the services we offer.

Wednesday, May 01, 2013

Outsource .NET Development


Are you exploring the possibilities to outsource your next .NET application? Beclasp Consulting offers a dedicated Offshore .NET Developers team to provide you best in class results through Microsoft Certified Developers. To know more please visit:

http://beclaspconsulting.net/outsource-dotnet-application-development

Thursday, May 17, 2012

The 'DbProviderFactories' section can only appear once per config file

Open machine.config at:
C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config\Machine.Config

Locate section:
<system.data>
<DbProviderFactories>
<!-- providers added here -->
<add name="... />
</DbProviderFactories>

Line to be removed to resolve the issue
<DbProviderFactories/>
</system.data>
done!







Tuesday, April 17, 2012

Could not load file or assembly 'Oracle.DataAccess' or one of its dependencies. An attempt was made to load a program with an incorrect format

Solution: In IIS Set 'Enable 32bit Applications' to True for the application pool used in the site.

If you are using ODP.NET in 64bit machine and have installed ODP.NET 32bit version you may get the error:

'Could not load file or assembly 'Oracle.DataAccess' or one of its dependencies. An attempt was made to load a program with an incorrect format'

To resolve the issue:
  • Select application pool that is getting used in the site.
  • Click on Advanced Settings...
  • Enable 32-Bit Applications - True
  • OK
  • You are done!

Monday, January 30, 2012

SQL Server 2012 (Denali): OFFSET and FETCH Limitations

OFFSET and FETCH can be used in any query that allows TOP and ORDER BY with the following limitations:
  1. The OVER clause does not support OFFSET and FETCH.
  2. OFFSET and FETCH cannot be specified directly in INSERT, UPDATE, MERGE, and DELETE statements, but can be specified in a subquery defined in these statements. For example, in the INSERT INTO SELECT statement, OFFSET and FETCH can be specified in the SELECT statement.
  3. In a query that uses UNION, EXCEPT or INTERSECT operators, OFFSET and FETCH can only be specified in the final query that specifies the order of the query results.
  4. TOP cannot be combined with OFFSET and FETCH in the same query expression (in the same query scope).

SQL Server 2012 (Denali) New Features: Limit number of rows using OFFSET and FETCH clause

Limiting the number of rows returned

The following example specifies an integer constant as the value for the OFFSET and FETCH clauses. The first query returns all rows sorted by the column ID. Compare the results returned by this query with the results of the two queries that follow it. The next query uses the clause OFFSET 5 ROWS to skip the first 5 rows and return all remaining rows. The final query uses the clause OFFSET 0 ROWS to start with the first row and then uses FETCH NEXT 10 ROWS ONLY to limit the rows returned to 10 rows from the sorted result set.

-- Return all rows sorted by the column DepartmentID.
SELECT ID, Name, GroupName
FROM Department
ORDER BY ID;

-- Skip the first 5 rows from the sorted result set and return all remaining rows.
SELECT ID, Name, GroupName
FROM Department
ORDER BY ID OFFSET 5 ROWS;

-- Skip 0 rows and return only the first 10 rows from the sorted result set.
SELECT ID, Name, GroupName
FROM Department
ORDER BY ID
OFFSET 0 ROWS
FETCH NEXT 10 ROWS ONLY;

SQL Server 2012 (Denali) New Features for Developers (Programming Enhancements)

Programmability enhancements in the Database Engine include FileTables, statistical semantic search, property-scoped full-text search and customizable proximity search, ad-hoc query paging, circular arc segment support for spatial types, support for sequence objects, default support for 15,000 partitions, and numerous improvements and additions to Transact-SQL.

Native XML Web Services (SOAP/HTTP endpoints) is Removed:
Beginning in Microsoft SQL Server 2012 Release Candidate 0 (RC 0), you can no longer use CREATE ENDPOINT or ALTER ENDPOINT to add or modify SOAP/HTTP endpoints.

FileTables:
The FileTable feature builds on top of the SQL Server FILESTREAM technology to bring support for the Windows file namespace and compatibility with Windows applications to the file data stored in SQL Server. This lets an application integrate its storage and data management components, and provides integrated SQL Server services (including full-text search and semantic search) over unstructured data and metadata, along with easy policy management and administration.

In summary, you can now store files and documents in special tables in SQL Server, but access them from Windows applications as if they were stored in the file system, without making any changes to the Windows applications.


Metadata Discovery:
The SET FMTONLY option for determining the format of a response without actually running the query is replaced with
sp_describe_first_result_set,
sp_describe_undeclared_parameters,
sys.dm_exec_describe_first_result_set, and
sys.dm_exec_describe_first_result_set_for_object.

EXECUTE Statement
The EXECUTE statement can now specify the metadata returned from the statement by using the WITH RESULT SETS argument.

A new family of supplementary characters (SC) collations can be used with the data types nchar, nvarchar and sql_variant.
For example: Latin1_General_100_CI_AS_SC or, if using a Japanese collation, Japanese_Bushu_Kakusu_100_CI_AS_SC.
These collations encode Unicode characters in the UTF-16 format. Characters with codepoint values larger than 0xFFFF require two consecutive 16-bit words. These characters are called supplementary characters, and the two consecutive 16-bit words are called surrogate pairs. SC collations can improve searching and sorting by functions that use the Unicode types nchar and nvarchar. For more information, see Collation and Unicode Support.
UTF-16 Supplementary Characters (SC) Collation Option for XML


Ad-hoc Query Paging Implementation
You can specify a range of rows returned by a SELECT statement based on row offset and row count values that you provide. This is useful when you want to control the number of rows sent to a client application for a given query.
 
SELECT ID, Name
FROM Person
ORDER BY ID
OFFSET 10 ROWS
FETCH NEXT 10 ROWS ONLY;

New Spatial Features
Circular Arc Segment Support for Spatial Types

Three new sub-data types for geometry and geography data types can be used to store circular arc segments, CircularString, CompoundCurve, and CurvePolygon. Methods for geography and geometry data types support the new circular arc segment data types. There are new methods for geometry and geography data types that work with circular arc segments, OGC Methods on Geography Instances, OGC Methods on Geometry Instances, Extended Methods on Geography Instances, Extended Methods on Geometry Instances and SQL MM Methods on Geography Instances. There are new static aggregate methods for geometry data type and geography data type, Extended Static Geography Methods and Extended Static Geometry Methods.

Sequence ObjectsA sequence object is a user-defined schema-bound object that generates a sequence of numeric values according to the specification with which the sequence was created. It operates similar to an identity column, but sequence numbers are not restricted to use in a single table.

THROW statementThe THROW statement can be used to raise an exception and transfer execution to a CATCH block of a TRY…CATCH construct.

14 New Functions and 1 Changed Function:

Microsoft SQL Server 2012 Release Candidate 0 (RC 0) introduces 14 new built-in functions. These functions ease the path of migration for information workers by emulating functionality that is found in the expression languages of many desktop applications. However these functions will also be useful to experienced users of SQL Server.

The new functions are:
Conversion functions    PARSE
    TRY_CONVERT
    TRY_PARSE
Date and time functions    DATEFROMPARTS
    DATETIME2FROMPARTS
    DATETIMEFROMPARTS
    DATETIMEOFFSETFROMPARTS
    EOMONTH
    SMALLDATETIMEFROMPARTS
    TIMEFROMPARTS
Logical functions
    CHOOSE
    IIF
String functions
    CONCAT
    FORMAT

In addition to the 14 new functions, one existing function has been changed. The existing LOG function now has an optional second base parameter.

SQL Server Express LocalDBSQL Server Express LocalDB is a new lightweight edition of Express that has all its programmability features, yet runs in user mode and has a fast, zero-configuration installation and short list of pre-requisites. The LocalDB edition of SQL Server is targeted to program developers. LocalDB installation copies a minimal set of files necessary to start the SQL Server Database Engine. To use the SQL Server Database Engine, developers connect directly to the database files and the SQL Server infrastructure is transparent to them. Developer Tools can provide developers with a SQL Server Database Engine that lets them write and test Transact-SQL code without having to manage a full server instance of SQL Server. An instance of SQL Server Express LocalDB is managed by using the SqlLocalDB.exe utility. SQL Server Express LocalDB should be used in place of the SQL Server Express user instance feature which is deprecated.

New and Enhanced Query Optimizer HintsThe syntax for the FORCESEEK table hint has been modified.
You can now specify an index and index columns to further control the access method on the index. The existing FORCESEEK syntax remains unmodified and works as before. No changes to applications are necessary if you do not plan to use the new functionality.

The FORCESCAN table hint has been added. It complements the FORCESEEK hint by specifying that the query optimizer use only an index scan operation as the access path to the table or view referenced in the query. The FORCESCAN hint can be useful for queries in which the optimizer underestimates the number of affected rows and chooses a seek operation rather than a scan operation. FORCESCAN can be specified with or without an INDEX hint.

Extended Event EnhancementsThe following new Extended Events are available.
page_allocated:
    Fields: worker_address, number_pages, page_size, page_location, allocator_type, page_allocator_type, pool_id
page_freed:
    Fields: worker_address, number_pages, page_size, page_location, allocator_type, page_allocator_type, pool_id
allocation_failure:
    Fields: worker_address, failure_type, allocation_failure_type, resource_size, pool_id, factor

The following Extended Events have been modified.
resource_monitor_ring_buffer_record:
    Fields removed: single_pages_kb, multiple_pages_kb
    Fields added: target_kb, pages_kb

memory_node_oom_ring_buffer_recorded:
    Fields removed: single_pages_kb, multiple_pages_kb
    Fields added: target_kb, pages_kb

   
OVER Clause Support Enhanced
The OVER clause has been extended to support window functions. Window functions perform a calculation across a set of rows that are in some relationship to the current row. For example, you can use the ROWS or RANGE clause over a set of rows to calculate a moving average or cumulative total.
In addition, ordering rows within a partition is now supported in the aggregate functions that allow the OVER clause to be specified.

Analytic Functions
The following analytic functions have been added.
CUME_DIST
LAST_VALUE
PERCENTILE_DISC
FIRST_VALUE
LEAD
PERCENT_RANK
LAG
PERCENTILE_CONT