Mining our knowledge Part 1

“The future success of companies and organizations will increasingly be based on their ability to unlock hidden intelligence and value from unstructured data, and text in particular”

(The 451 Group, 2005)

The Information Age has made organisations data rich, instead of being starved of information; they now have volumes of data available to them. This presents challenges because organisations often lack the resources to convert them into useful intelligence.

Amongst this data is qualitative data i.e. a person’s opinion of a product or a service, which can be far more useful in developing insight instead of hard facts and figures. However translating qualitative data into knowledge is really hard work. This is because this information is usually collated in a text form made up of observations and quotes, and it takes a lot of organisational resources including time, people and technology to develop it into useful intelligence.

One method available to organisations to develop this knowledge is text mining. Text mining has traditionally been associated with the academic and research fields, but now its becoming more widespread with commercial organisations recognising that the unstructured data that they hold is as valuable as their structured data since it offers a more holistic view of the organisation. As a result many companies are offering text analytics, the text mining equivalent used in business settings as a solution for analysing this data (Feldman, 2004).

So what is text mining?

Text mining is “the discovery by computer of new, previously unknown information, by automatically extracting information from different written resources… linking together the extracted information together to form new facts or new hypotheses…” (Hearst, 2003).

The key tasks of text mining include:

·         Classification - The process of tagging new data based on its features.

·         Clustering - The process of grouping documents based on the similarities in the content

·         Association - The process of organising information into hierarchical networks. (Leong et al., 2004)

Transforming unstructured text into intelligence

Qualitative data can be taken from many different sources including emails from customers, employees and suppliers, research reports. It can also be published information about competitors such as promotional materials & customer comments, and information related to legislative and regulatory changes (Leong et al, 2004).

This data is turned into a structured and a relevant form. The “mining process” is made up of three elements:

1. “information selection and preprocessing;

2. patterns analysis, recognition and visualization; and

3. validation and interpretation” (Zhang and Segall, 2010, p.625).

Mining this information enables organisation to enhance their business intelligence (BI). Wang and Wang (2008, p.623) state that “the central theme of BI is to fully utilize massive data to help organizations gain competitive advantages”. BI helps organisations to discover the knowledge hidden in their data.

Text mining (TM) helps organisations to develop knowledge using “various algorithms and tools to extract metadata or high-level information and/or to discover patterns and relationship within the extracted information” (Choudhary et al. 2009, p.730). The subsequent knowledge can help decision makers to make more informed decisions.

The difference between data, text and web mining

Zhang and Segall (2010 p.625) highlight the differences between data, text and web mining stating that “Data mining primarily deals with structured data organized in a database. Text mining most handles unstructured data/text. Web mining lies in between and copes with semi-structured data and/or unstructured data”.

Ok so that's part 1, in part 2 I’ll outline  the ways in which commercial organisations are using text mining to move their business forward. See you next time!

Note: I developed this post a while ago, initially as part of an essay for the MMU Information and Communications department.

References

CHOUDHARY, A.K., OLUIKPE, P.I., HARDING, J.A. and CARILLO, P.M., 2009. The needs and benefits of text mining applications on Post-Project Reviews. Computers in Industry[online]. 60 [cited 12 March 2011] pp.728-740.

FELDMAN, R., 2004. Text Analytics: Theory and Practice, ClearForest Corporation. [online] Available at: [cited 27 March 2011].

HEARST, M., 2003. What is text mining?. Berkeley, [online] Available at: [cited 2 March 2011].

LEONG, E.K.F., EWING, M.T. and PITT, L.F., 2004. Analysing competitors’ online persuasive themes with text mining. Marketing Intelligence & Planning [online]. 22 (2) [cited 14 March 2011] pp. 187-200.

THE 451 GROUP, 2005. Text-aware Applications: The Endgame for Unstructured Data Analysis. cited by CLARABRIDGE, 2008. Text Mining’s Moment: The three trends triggering commercial adaptation. clarabridge.com, [online] Available at: [cited 1 March 2011].

WANG, H. and WANG, S., 2008. A knowledge management approach to data mining process for business intelligence. Industrial Management & Data Systems [online]. 108 (5) [cited 12 March 2011] pp. 622-634.

ZHANG, Q. and SEGALL, R.S., 2010. Review of data, text and web mining software. Kybernetes [online]. 39 (4) [cited 1 March 2011] pp.625-655.

Customer Intelligence: The game changer - Part 2

In my last post I defined what customer intelligence is, its purpose & benefits. This post is about how to develop customer intelligence for the organisation. 

Customer intelligence! How do we do it?

Developing good customer intelligence takes time, money and effort, but in exchange the sizeable benefits for organisations include better streamlining of services, better allocation of resources, more tailored products and improved customer satisfaction levels. But how do we develop this intelligence?

Data can be developed intelligence across four stages. I've illustrated the journey in the following model:

Each stage presents a number of questions and actions that an organisation has to answer and to do respectively to develop its knowledge.

Stage 1: Set out your customer intelligence objectives

This stage involves establishing why you are developing customer intelligence. As I’ve mentioned before, developing customer intelligence can be a costly, resource heavy and usually an extremely lengthy exercise. This stage involves developing a business case for this work, to justify its need for the organisation. This is the also the stage where the buy-in is sought from the various stakeholders, including the organisation as a whole especially the top tier, the users and suppliers. Ensure that you are clear as an organisation about why you are developing customer intelligence. I've jotted down a few questions that you may want to ask at this stage:

• Have you got a new product that you want to launch?
• Are you trying to improve an existing service?
• Are you trying to improve your market share?
• Is there a legislative/regulatory requirement?

Stage 2: Identify the data needed to create information

Sometimes organisations can spend so much time and energy on setting out its objectives and resourcing customer intelligence projects, that the most obvious needs of the organisations can be overlooked. Often the right data isn't collected, whether that’s missing out vital data or asking non relevant questions.

• Be clear about data you need to develop your customer intelligence. 
• Consult with the different departments in your organisation about what their customer intelligence needs are and what data do they need to develop it.
• Have somebody senior within the organisation who can approve (and veto) what data is collected. It’s not always possible to collect everything that everyone needs; otherwise the data collation exercise would become unmanageable.
• Remember the data will need to be processed into a structured form to convert it into information to meet your objectives.  Establish how you are going to do this, who will collect the data, who will convert it into information, what systems will you use.

Stage 3: Evaluate the information to develop knowledge

Evaluate the information to develop the customer intelligence. Use it to reflect on what you have learnt:

• What is it telling you about your customers? 
• What actions do you need to undertake based on the information to move closer to meeting your objectives?

Stage 4: Act on the customer Intelligence and evaluate the subsequent results

Take action based on your customer intelligence and evaluate the results

• What were the outcomes of your actions?
• Did they help you meet your objectives?
• Did the customer intelligence and subsequent actions yield the results that you expected?
• Did it reveal a need for developing new objectives or re-evaluating existing ones?

Customer intelligence costs organisation's time, money and effort to develop, so it’s important to maximise its success and justify its existence.Ok well that’s it for today..... 

References

GRIMES, N., 2009. The nine steps to best practice customer insight, My Customer.Com. [online] Available at: <http://www.mycustomer.com/topic/customer-intelligence/best-practice-insight-what-essential-invisible-eye> [cited 30 October 2011].

Customer Intelligence:The game changer - Part 1

The phrase 'know your customer’ is an often repeated mantra, in today’s business world. It’s a phrase that’s uttered in high level meetings and in weekly performance appraisals. The point is, that it’s vital for organisations to ‘know who their customers are’ and ‘what they need or want’, to reasonably tailor their services to meet those requirements. 

As customers ourselves, we ‘enjoy’ elements of personalisation in the services that we receive, whether that’s through Tesco’s club card scheme, personalised recommendations based on previous purchases from Amazon or being able to like companies on Facebook which push news to us in our activity streams. We are getting used to this type of service and the core enabler behind this is customer intelligence. It’s also known as customer insight and customer knowledge.

We are living in the Information Age according to the academics. We've gotten data rich because of the web, we even contribute to it by generating content  even to the point that we have to mine our data to make sense of it (a future post). We now have the tools to learn more about our customers but so do our peers or competitors. If we fail to understand to what our customers need or want, they will go to somebody that does. (check out my post on how all of us are making the web better)

This applies to the social housing sector as well. On the face of it, demand for social housing exceeds supply, which limits customer choice but tenants aren't the only customers. Stakeholders like the Homes and Communities Agency and the Banks are also our customers, we need to demonstrate that we are delivering good tailored services to our tenants to engage these stakeholders trust and confidence.  What’s at stake? Well a simplistic answer is organisational growth.  Reduced backing from these bodies would result in insufficient resources for developments and stock transfers. (Check out my posts on what social housing is and a brief history of social housing to find out more about the sector).

What is customer intelligence?

Customer intelligence is the process of using relevant information to develop a better understanding of customer preferences, beliefs and aspirations to enable the design and delivery of better and more personalised products and services.

Customer intelligence can help organisations to:

• Streamline its processes for delivering their products and services based on a better understanding of customer beliefs and expectations.
• Identify their priorities to enable the easier allocation of resources
• Develop better tailored products and services based on the customers requirements.
• Improve customer service levels and increase satisfaction rates

Above all else customer intelligence should be used by organisations to inform their strategic planning and should be used as tool in the decision making process.

Two pitfalls lot of organisations fall into, include treating the customer intelligence process as just data collation exercise, forgetting that data needs to be developed into useable knowledge for the organisation. Customer intelligence isn’t about collecting data, it’s about what you intend to do with it. The other thing organisations need to watch out for is treating the customer intelligence as a one off process; it’s a continuous activity which helps organisations to grow its knowledge of its customers. Remember, customer needs and preferences change, customers change for that matter (IDEA, 2009).

Creating Intelligence

The process of developing customer intelligence, in my opinion is linked to knowledge development within organisations. Customer intelligence begins life as unstructured raw data gathered through consultations, surveys and a variety of other methods. The data is contextualised and organised in relation to the organisations targets or key performance indicators to organise it into facts and figures to convert it make it useable information. Knowledge is created when the information and is considered & analysed in relation to the organisations ethos and goals. Previous experiences are applied to the information by the organisation to reflect and improve itself and this is the stage where simple information becomes organisational knowledge.

The following diagram outlines this process:

The process of converting data into information and transforming information into organisational knowledge

This is the end of part 1, in the next instalment, I will propose a few questions that organisations should ask themselves when they are developing their customer intelligence. I will also outline the key elements to consider when developing customer intelligence.  See you next time!

References

IDEA, 2009. What is customer or citizen insight? [online] Available at: <http://www.bbc.co.uk/news/uk-14380936> [cited 30 October 2011].

A brief history of social housing

Carrying on from my previous post about what is social housing?, I'm doing this follow up on how social housing has developed over the years. 

A whistle stop tour of social housing

Social housing has existed in England since 1890’s, in fact much longer before then in one form or another. It really took off with the Housing and Town Planning Act in 1919 making it law for Local Authorities to provide council housing. The Liberal Democrat government at the time initiated the first sustained effort to provide low cost affordable housing. They set a highly ambitious target to build 500,000 homes and failed miserably building only around half this number during their time in power. The subsequent Conservative and Labour governments shifted the focus on to private housing. Home building didn’t stop in the period up to the second world war with another 580,000 homes built, but it was after the second world war that social housing really soared with over a million homes built, four fifths of which were ‘council housing’(Shapely, n.d.; Wheeler, 2011).

The subsequent decades saw a shift towards replacing slum housing, rehousing people into ‘gleaming’ high rise towers and out of town overspill estates spurred on by the lack of land available in the central council areas. This new promised land turned out to be pretty bleak with problems rooted in poor design and substandard construction, as well as a lack of consultation about what people wanted by the town planners (Shapely, n.d.; Wheeler, 2011).

This changed in the late sixties, with council tenants becoming frustrated with the poor housing conditions and with their opinions overlooked began to organise themselves. Tenant groups sprang up in different parts of the country to object to the housing conditions being imposed on people, with limited success. Ultimately the Local Authorities realised that they were losing public confidence and as a result, a few began to develop tenant participation schemes with the very first tenant participation handbook developed in the mid seventies (Shapely, n.d.).

The Eighties ushered a resurgence in social housing, with the Conservative government, exploring alternatives to council housing, supporting initiatives with grants and encouraging private finance in the sector. This was the ideal opportunity for not for profit housing associations to invest into the sector by tapping into finance not traditionally available to the government This period also saw ‘right to buy’ take off with over 2 million council homes sold to tenants from the 1980’s to the present day. This sale of council housing highlighted the need for increased social housing, with housing waiting lists feeling the strain. The 1998 Housing Act helped create the present system, supporting housing associations in attracting investment and making it easier for them to collect rent for people entitled to housing benefit (Heywood, 2010).

The two decades since have seen local authorities transfer council housing to large scale voluntary trusts to attract private finance to improve stock. In this period Housing associations have attracted finance to develop new homes and improve existing ones (Heywood, 2010).

The sector has experienced frequent changes in its regulatory regime over the past few years with the introduction of a new regulatory framework and the decommissioning of two principle housing bodies; the Audit Commission and the Tenant Services Authority by December 2011. This affects how RSL’s will be regulated in the future, “with the arrangements for housing inspections still unclear” (Inside Housing 2010).

The announcements in the comprehensive spending review announced in October 2010 have also affected the sector, with RSL’s capital budgets for affordable house building reduced by 60%. This means that 7,000 less homes will be built over the next four years (Inside Housing 2010; Ropke 2010). Changes to the housing benefit system coming into force in 2013 are also expected to affect RSL’s incomes because a high number of social housing tenants are economically disadvantaged and in receipt of housing benefit (Audit Commission, 2010; Parliament UK).

Like i said this was a brief look at social housing has developed to what it is today, feel free to discuss. You may want to have a look at one of my previous posts social housing 2.0, which outlines the direction I think the sector should be looking to move towards to improve its image. See you next time, when I'm starting my series of continuous improvement posts starting with customer intelligence.

References

AUDIT COMMISSION, 2010. Housing Inspections. [online] Available at: <http://www.audit-commission.gov.uk/housing/inspection/pages/default.aspx> [cited 30 October 2011].

HEYWOOD, A., 2010, Investing in Social Housing, A Guide to the Development of the Affordable Housing Sector, The Housing Finance Sector [online] Available at: <http://thfcorp.com/investing/THFC%20-%20Investing%20in%20Social%20Housing%20230510.pdf> [cited 30 October 2011].

INSIDE HOUSING, 2010. The year that changed social housing forever. [online] Available at: <http://www.insidehousing.co.uk/> [cited 30 October 2011].

PARLIAMENT UK. 2010. Impact of the changes to Housing Benefit announced in the June 2010 Budget. [online] Available at: <http://www.publications.parliament.uk/> [cited 30 October 2011].

SHAPELY, P., N.D., Social housing and tenant participation. [online] Available at: <http://www.historyandpolicy.org/papers/policy-paper-71.html>

WHEELER, B., 2011, What future for social housing?, BBC [online] Available at: <http://www.bbc.co.uk/news/uk-14380936> [cited 30 October 2011].

What is social housing?

I was halfway through working on my 'surveys' post on Digibored, when I realised that I’ve not actually described the social housing sector despite mentioning it frequently in this blog. So I’m course correcting and giving a short insight into what the social housing sector is. Digibored is about using technology and communication to improve and develop organisations. And I think it’s important to contextualise these topics, so I’m course correcting to set the scene for this blog.  

This post is a brief outline of what social housing is and I will be doing a follow up post on the history of social housing. Consider them my prequels to my earlier post called social housing 2.0.

So, what is social housing?

Simply put, social housing is housing usually owned by councils, housing association or other not for profit organisations such as cooperatives. These organisations provide affordable rental accommodation to vulnerable people and those on a low income (Directgov, n.d.; BBC, 2007).

Almost 1 in 5 people in England live in social housing. Traditionally social housing customers have experienced limited mobility due to demand for social housing exceeding supply. Because there is so much demand for social housing, it’s allocated according to need. Preference is given to the more vulnerable sectors of our society with a sliding scale used to rate housing need. The more points an applicant has on the rating system, the higher they are on the ‘housing list’ (Local Governments Association 2008; Shelter, 2011; Survey of English Housing 2008)

Who owns it?

Social housing is often perceived as ‘council housing’ by the average person, a point I picked up on in social housing 2.0, but in reality social housing is owned and managed by several different types of organisations. For example in central Manchester, there are 67,420 social housing properties. The following table highlights the different types of companies that own and manage these properties (stats taken from Housing Net)

Type of Organisation	Number of Properties Owned
Arms Length Management Organisations	29,029
Charities or Charitable Organisations	224
Co-operatives	203
Housing Associations	37,779
Other	185
Total	67,420

The majority of the stock is owned by Housing Associations and Arms Length Management Organisations, so this is the bit where I tell you what they are:

Housing Associations

Housing Associations also known as resident social landlords are usually not for profit organisations that own and manage properties acquired through local authority housing stock transfers and by developing properties. (Directgov).

Arms Length Management Organisations (ALMO’s)

ALMO’s are organisations created by local authorities to manage and improve its housing stock. The local authorities retain the ownership of the properties and the ALMO looks after the day-to-day management of the properties dealing with issues like repairs and rental queries.

So this is it! A very brief look at what social housing is. It’s a very broad subject and I’ve only tried to include things that help set a context for the blog. As I mentioned earlier, the next post is about the history of social housing… see you on the flip sideJ.

References

BBC, 2007, Q&A: Social housing. [online] [retrieved 28 October 2011] <http://news.bbc.co.uk/1/hi/uk_politics/6691927.stm>

 

COMMUNITIES & LOCAL GOVERNMENT, 2009. Survey of English Housing Preliminary Results 2007/08. [online] [retrieved 28 October 2011] <http://www.communities.gov.uk/publications/corporate/statistics/sehprelimresults0708>

DIRECTGOV. N.D., Housing Associations – what are they. [online] [retrieved 28 October 2011] <http://www.direct.gov.uk/en/HomeAndCommunity/Councilandhousingassociationhomes/Housingassociationhomes/DG_188384>

LOCAL GOVERNMENTS ASSOCIATION, 2008. COUNCILS AND THE HOUSING CRISIS: The potential impacts and knock-on effects of the credit crunch on councils and their housing role [online] [retrieved 28 October 2011] <http://www.lga.gov.uk/lga/aio/569196>

SHELTER, 2011, What is social housing,[online] [retrieved 28 October 2011] <http://england.shelter.org.uk/campaigns/housing_issues/Improving_social_housing/what_is_social_housing>

8DZFQ5F48GNB

STOCKPORT HOMES, 2010, What is an ALMO, [online] [retrieved 28 October 2011] <http://www.stockporthomes.org/main.cfm?type=WHATISANALMO>

How all of us are changing the web: Part 2

In my digibored post 'How all of us are changing the web: Part 1', I talked about User Generated Content (UGC) and Crowdsourcing and how we have used them to change the web as we know it today. This is a follow up; I’ve picked out a few common ways that we use UGC and crowdsourcing. There are plenty of others, but for digibored I only need a few to highlight my points. If you want a fuller list, wiki the terms.

1. Ratings and Reviews - UGC
2. Tagging Stuff - UGC
3. Citizen Science - Crowdsourcing
4. Citizen Media - Crowdsourcing

Ratings and Reviews

This is probably one of the most common forms of UGC. A lot of us shop online (figure here). I know personally, every time I make a purchase on Amazon or Ebay, I’m prompted to rate and review what I’ve bought. I know I often check user ratings because it’s damn confusing with the sheer number of similar products on offer. And it’s just not shopping when I use my local public library service, reserving books online, again I have the option to rate and review my loans. So what we think of our purchases are a vital resource in the online marketplace.

For example, a study by Comscore revealed that online user-generated reviews influence one in four of us when we are purchasing a product or service online (2007). That makes our ratings, our reviews very influential, in fact Williams & Co. add to this saying that:

1. “Research has shown that many people trust other people’s reviews more than they trust expert opinion”

2. “There is a strong relation between positive reviews of a product or service and satisfaction with sales” (2010, p118).

Tagging Stuff

Have you ever tagged something online? No, are you sure?

What about tagging people in one of your pictures on Facebook? Or how about retweeting posts and adding hashtags to entries, on Twitter? If you have done, then, you’ve generated content online.  

We can assign keywords to various content, whether that’s a document on a intranet, a blog, a picture or a video that we’ve uploaded. This started off as a metadata tagging where we could choose a preset word in an online database to describe an item to help it be found again when we search for it. It’s evolved into folksonomy, where we can tag something with our own words which is what we tend to on platforms like Flickr and YouTube (Vander Wal, 2007).

Point is that tagging helps us to find things easier online and in the case of twitter and other social networking sites, it allows us to recycle and reshare content (Huang et al., 2010).

Citizen Science

What is it about the worked ‘citizen’ it just makes me think of worker ants…

Talking of worker ants, the science research sector has successfully tapped into crowdsourcing to help create citizen science. We all get a chance to be scientists finally; putting to use that home lab kit you got as a present from your mum or dad when you were nine. Well, you get to use for something other than blowing up your bedroom …Sorry Mum…  

Anyway back to the topic, ‘Citizen Science’ is about the average person helping with research on a range of scientific subjects. For example, in some companies are using services Innocentive to place scientific challenges to an open audience and awarding a prize fee for the best solutions. We can also contribute by sharing our spare computing power to help create a free online open-source lab for researchers for things like cancer research etc. by downloading World Community Grid Software (Howe, 2006; The Marketer, 2011).

      Citizen Journalism

There’s that ‘citizen’ word again. This time it’s being used by the media. ‘Citizen Journalism’ has become an extremely popular and vital resource in today’s day and age. It lets us report events as they happen. Essentially, citizen journalism involves “a wide range of activities in which everyday people contribute information or commentary about news events” (Educause, 2007).
     

Citizen journalism is used by major news providers like the BBC and CNN, as well as amateur news sites such as Cyber Journalist as well as blogs and video sharing sites. In fact Twitter posts are the main source of information via the Google real-time service (Educause, 2007, Google, N.D.).

Richard Sambrook, a BBC journalist summed up the benefit of citizen journalism in 2009 saying that “when major events occur, the public can offer us as much new information as we are able to broadcast to them. From now on, news coverage is a partnership”

Lets sum up

I’ve told you four ways that I think have helped us all to change the web, its nots a complete list, its not meant to be. In fact UGC and crowdsourcing aren’t the only ways that we are making web better, but like I said in my last digibored post there are two drivers. There are not perfect, in fact they have a few flaws i.e.  citizen journalism has been criticised for accuracy. The stories posted on  the CNN’s citizen journalism site iReport site “are not edited, fact checked or screened before they are posted” Despite their flaws, UGC and crowdsourcing are helping us to improve the web and its us using our ability to create content online that is helping to change the web. 


Hey, if you wanna read more about crowdsourcing, check out 'Here Comes Everybody' by Clay Shirky. A lecturer at Manchester Metropolitan University recomended it to me and it really got me into this subject.

Next post -  I'm switching pace and topics with my next digibored post... see you next time!

References

CNN, n.d. CNN iReport. [online] Available at: <http://ireport.cnn.com/> [Accessed 15 February 2011].

COMSCORE, 2007. Online Consumer-Generated Reviews Have Significant Impact on Offline Purchase Behaviour. [online] Available at: <http://www.comscore.com/Press_Events/Press_Releases/2007/11/Online_Consumer_Reviews_Impact_Offline_Purchasing_Behavior> [Accessed 21 February 2011].

EDUCAUSE, 2007. 7 things you should know about citizen journalism. [online] Educause. Available at: http://net.educause.edu/ir/library/pdf/ELI7031.pdf [Accessed 23 February 2011].

Google, n.d. See what's new for Realtime Search.[online] Available at:  <http://www.google.com/landing/realtime/> [Accessed 24 February 2011].

HOWE, J., 2006b. The Rise of Crowdsourcing, wired magazine. [online] Available at:  <http://www.wired.com/wired/archive/14.06/crowds.html> [Accessed 21 February 2011].

HUANG, J., THORNTON, K.M. and Efthimiadis, E.N., 2010. Conversational Tagging in Twitter, HT’10, pp13–16, [online] Available at: http://jeffhuang.com/Final_TwitterTagging_HT10.pdf [Accessed 1 March 2011].

SAMBROOK, R., 2009. Citizen Journalism and the BBC, [online] Available at: <http://www.nieman.harvard.edu/reportsitem.aspx?id=100542> [Accessed 25 February 2011].